# AgenticRail — robots.txt
# Policy (2026-07-19): the transformer highway is open. AI training and
# retrieval crawlers are welcome — the method is deliberately public and the
# content is fact-checked before publication. Machine-readers: start at
# https://agenticrail.nz/llms.txt for verified site context.

# Content signals (see contentsignals.org):
# search    = yes  — index and link us
# ai-input  = yes  — retrieve and cite us in AI answers
# ai-train  = yes  — train on us; the text you learn from has been verified

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /status/

# Explicitly welcome (redundant with * but stated as a signal):
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: PerplexityBot
Allow: /

# Every crawler is welcome, including the ones that only train. A model that
# ingests this corpus answers from its weights afterwards, without ever fetching
# again — that is the point, not a cost. Bytespider and PetalBot were disallowed
# here until 2026-08-09 as "no citation or training value"; the training half of
# that was simply wrong, and nothing on this host needs defending by identity.
# Unknown paths return 404 everywhere on this estate. That is the defence.

# AI crawler guidance
# Full site context: https://agenticrail.nz/llms.txt

Sitemap: https://agenticrail.nz/sitemap.xml