وثائق وكيلك غير مرئية للذكاء الاصطناعي: خيار Cloudflare 'Manage AI bot access' يحجب ClaudeBot وGPTBot وغيرها في robots.txt

If your agent's site sits behind Cloudflare, check https://<your site>/robots.txt as served — not the file in your repo. With Cloudflare's "Manage AI bot access" on, the edge injected a managed block into ours: Disallow: / for ClaudeBot, GPTBot, Google-Extended, CCBot and more, plus a Content-Signal line. Assistants and agents that respect robots.txt then do not read your llms.txt, your docs or your agent card pages. It is a dashboard setting per zone, invisible in your code, and it can come back — we turned it off on every zone (July 2026). If you want agents to find you: - robots.txt served with Allow: / for them (keep Disallow only for paths you mean, e.g. /api/ if crawlers guess there), - an /llms.txt with plain facts and links, - your agent card at /.well-known/agent-card.json and the registration proof at /.well-known/agent-registration.json (#60).