robots.txt + llms.txt + AI-Bot Policy Builder

llms.txt is an emerging convention (spec at llmstxt.org) — a Markdown file at your site root (e.g. https://yoursite.com/llms.txt) that gives LLMs a curated index of your most important pages plus a short summary. It's like robots.txt for LLMs, but instead of telling crawlers what NOT to do, it tells them what to read first so they cite you accurately. You don't strictly need it yet — most LLMs still crawl HTML — but adopting early is cheap insurance for AI-search visibility, the same way XML sitemaps were optional for years before becoming standard.

Allowing AI crawlers is how you become a citable source inside ChatGPT, Claude, and Perplexity answers. When a user asks an AI assistant a question your content can answer, the assistant pulls from sites it has crawled and cites them by name with a clickable link. If you block these bots you become invisible in AI answers — even if you rank #1 in Google. For most marketing sites, the citation traffic is worth more than any copyright concern.

Blocking dedicated AI bots (GPTBot, ClaudeBot, CCBot, etc.) does NOT affect Google Search, Bing, or any traditional SEO ranking — those use different user-agents (Googlebot, Bingbot) which you should never block. Special case: blocking Google-Extended only opts you out of Gemini training, NOT Google Search. Same for Applebot-Extended (opts out of Apple Intelligence training, doesn't touch Spotlight or Siri). The only SEO-adjacent risk is losing visibility inside AI search engines (ChatGPT search, Perplexity, You.com).

No. robots.txt is a voluntary standard (RFC 9309). Well-behaved crawlers from OpenAI, Anthropic, Google, Microsoft, and most reputable AI companies do respect it. Bad actors — scrapers, training datasets that anonymise their fetches, or unbranded crawlers — ignore it entirely. For hard enforcement use server-side rules: Cloudflare bot management, WAF rules, IP allowlisting, or rate-limiting at the edge. Treat robots.txt as a polite request, not a security control.

Yes — the robots.txt spec supports per-user-agent Disallow paths. The current builder generates a site-wide Allow / Disallow per bot for simplicity. Per-bot path-level rules (e.g. block GPTBot from /pricing only) is on the roadmap. If you need it now, hand-edit the generated file: under the GPTBot block, replace "Disallow: /" with one or more "Disallow: /pricing" lines.

Both files go at your site root, served at the URL paths /robots.txt and /llms.txt with content-type text/plain. On Next.js you can put them in /public, return them from app/robots.ts and app/llms.txt/route.ts, or generate them at build time. On WordPress, most SEO plugins (Yoast, Rank Math, AIOSEO) have a robots.txt editor. On a static site, just commit the file. Don't forget to ping Google Search Console to re-crawl after major robots.txt changes.