# AI Search Crawlers (Explicitly Allow) # NOTE: robots.txt user-agent groups do not merge — a bot matches only its own # most-specific group, never falling back to combine with the `User-agent: *` # rules below. Each group here repeats the same Disallow set as the wildcard # group so admin/auth/API paths stay excluded from every crawler, not just # unnamed ones (found in the M10 audit; applies site-wide, not just to /ci). User-agent: GPTBot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: anthropic-ai Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: PerplexityBot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: YouBot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: meta-externalagent Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ User-agent: Diffbot Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ # Training/Scraping Crawlers (Block) User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: / # Social media link-preview crawlers (need full page access to read OG tags) User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot Allow: / User-agent: facebookexternalhit Allow: / # Standard Crawlers User-agent: * Allow: / Disallow: /app/ Disallow: /login Disallow: /signup Disallow: /setup Disallow: /api/ # LLM-readable site summary # https://www.intelcue.ai/llms.txt Sitemap: https://www.intelcue.ai/sitemap.xml