# Allow search engines User-agent: * Allow: / Disallow: /*/dashboard Disallow: /*/login # MDMENU1 (2026-10-23): every public menu also has a plain-text (Markdown) # edition at //menu//md for AI assistants. It is a public page # and no rule below blocks it; the `*` group above allows it. # OGCPU1 (2026-10-08): OG image routes rendered per fetch (937 ms MISS each, # Search Console 5xx on /{lang}/{vertical}/opengraph-image + blog). They are # static now; keep crawlers off them anyway. Social scrapers (X/Facebook/ # WhatsApp) fetch og:image regardless of robots.txt. No twitter-image routes # exist (verified 2026-10-08), so no twitter-image line is needed. Disallow: /*/opengraph-image Disallow: /opengraph-image # --------------------------------------------------------------------------- # SEARCH + RETRIEVAL AGENTS — explicitly allowed everywhere. # # 🔴 Added 2026-08-29 (master-order row 8). These are NOT training scrapers, and # two of them were being blocked as if they were. Each role below is quoted from # that vendor's own crawler documentation, not from memory: # # OAI-SearchBot "used to surface websites in search results in ChatGPT's # search features" (developers.openai.com/api/docs/bots) # ChatGPT-User "used for certain user actions in ChatGPT ... When users ask # ChatGPT ... a question, it may visit a web page" (same) # Claude-SearchBot "navigates the web to improve search result quality" # Claude-User "supports Claude AI users. When individuals ask questions to # Claude, it may access websites" (support.claude.com/en/articles/8896518) # PerplexityBot "designed to surface and link websites in search results on # Perplexity. It is NOT used to crawl content for AI # foundation models" (docs.perplexity.ai/guides/bots) # Perplexity-User "supports user actions within Perplexity" (same) # # Why this matters: we publish a per-merchant discovery site at /s/ — # indexable dish pages, LocalBusiness/Menu/MenuItem/Offer/AggregateRating # JSON-LD, opening hours, a sitemap. That is the whole point of it. Blocking the # retrieval agents meant a diner asking an assistant "where should I eat # tonight" could never reach a merchant's page, from our own file. # --------------------------------------------------------------------------- User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --------------------------------------------------------------------------- # AI TRAINING CRAWLERS: every public page (founder decision 2026-10-15: "we let # ai training bots read every public page obviously"). # # They have NO group of their own, on purpose, so each one follows # `User-agent: *` at the top: every public page, never the dashboard, the login # or the OG-image routes. A group naming a bot REPLACES the `*` rules for that # bot (RFC 9309), so a group added here must copy those Disallow lines. # # Until 10-15 these were limited to /for-ai-assistants, /llms.txt and /blog: # GPTBot, CCBot, anthropic-ai, Claude-Web, ClaudeBot, Google-Extended, # Bytespider, Amazonbot, FacebookBot, Omgili, Diffbot, Applebot-Extended, # YouBot, Cohere-ai. Cloudflare also answered 403 to the training crawlers # until the founder set its AI bot policy to Allow on 10-15. # # Why: a model trained on these pages can name QRDERS later, including every # merchant's /s/ page. The cost is bandwidth (OpenAI's crawlers read about # 500 MB of HTML in the 10-15 Cloudflare export). If one bot hammers the site, # block THAT bot in Cloudflare, not all of them here. # --------------------------------------------------------------------------- # Sitemap Sitemap: https://qrders.com/sitemap.xml