# KreatorMesh — crawl policy
#
# AI assistants are a deliberate acquisition channel (docs/ai-distribution-plan.md), so
# AI crawlers are allowed in on purpose. Content-Signal states that permission explicitly:
#   search=yes    may index this for search results
#   ai-input=yes  may read this to answer a user's question right now
#   ai-train=yes  may train on this
# Setting ai-input=no would keep KreatorMesh out of AI answers entirely.
#
# NOTE: this file is not the only gate, and on 2026-09-05 it was not even the effective one.
# Two Cloudflare zone settings were overriding it and have since been turned off:
#   ai_bots_protection: block   -> 403'd ClaudeBot/PerplexityBot at the edge
#   is_robots_txt_managed: true -> PREPENDED "Disallow: /" for ClaudeBot, GPTBot,
#                                  Google-Extended, CCBot etc. above these rules
# The second one is the dangerous one: edits here deploy fine and do nothing, because
# crawlers honour the first matching group. If AI traffic ever disappears, check the zone
# config before editing this file. Verify with:
#   curl -sI -A "ClaudeBot" https://kreatormesh.com/ | head -1        # expect 200
#   curl -s https://kreatormesh.com/robots.txt | grep -cE "^User-agent: (ClaudeBot|GPTBot)"   # expect 0

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /admin
Disallow: /reviewer-access

Sitemap: https://kreatormesh.com/sitemap.xml

# Machine-readable index of the programmatic entry points (MCP server, OpenAPI, agent plugin).
# Same convention post-bridge uses; agents that look for it find the MCP server without scraping.
Agentmap: https://kreatormesh.com/.well-known/ai-catalog.json