llms.txt vs Robots.txt vs Sitemap: What Each File Does for AI Crawlers

People keep treating robots.txt, sitemap.xml, and llms.txt as three versions of the same thing. They are not. One controls access, one lists your pages, and one — the newest — suggests your best content to AI assistants. Mix them up and you either block the crawlers you want or waste effort on a file that does nothing for you. This post explains exactly what each file does, which AI crawlers actually honour which, and the minimal correct setup for an Indian B2B site in 2026.

What Does Each File Do, in One Line?

Robots.txt tells crawlers which paths they may or may not fetch. Sitemap.xml gives crawlers a complete list of the URLs you want indexed, with last-modified dates. Llms.txt is a curated Markdown file that points AI assistants to your most important, cleanest content so they summarise you accurately. Access control, URL inventory, and AI curation — three separate jobs, three separate files.

Why Does This Confusion Matter Now?

AI search is now a real traffic and citation channel. Roughly 40% of AI-generated answers include citations to source pages, and the sites that get cited tend to be the ones AI engines can crawl cleanly and parse easily. If your robots.txt accidentally blocks an AI crawler, you are invisible to that engine. If your sitemap is stale, new pages take longer to be discovered. And llms.txt, proposed by Jeremy Howard in September 2024, is being added to sites on the bet that a clean content map helps assistants quote them correctly.

The risk is doing the wrong one. We have audited Indian B2B sites that spent a week crafting an llms.txt while their robots.txt quietly blocked GPTBot. The file that mattered was the one they ignored.

How Do the Three Files Compare?

The fastest way to keep them straight is side by side. Here is what each does, where it lives, and who reads it.

What Does robots.txt Actually Control?

Robots.txt is an access-control file. It sits at your site root and lists rules per user-agent: which paths a given crawler may fetch and which it must skip. It is the oldest of the three — proposed in 1994 and formalised as RFC 9309 in 2022. It is a request, not a hard wall; well-behaved crawlers obey it, but it does not stop a determined scraper.

For AI specifically, robots.txt is where you allow or block named AI crawlers. The agents you most likely care about:

What Does sitemap.xml Actually Do?

A sitemap is an inventory. It is an XML file listing every URL you want indexed, ideally with a lastmod date so crawlers know what changed and when. It does not control access — that is robots.txt's job — and it does not guarantee indexing. It speeds discovery, especially for new or deep pages a crawler might not reach by following links.

The most useful field for AI and search alike is lastmod. Freshness is a signal: content that has not been updated tends to decline in AI-citation frequency over time. A truthful, accurate lastmod helps crawlers re-fetch your updated pages promptly. Do not fake it — bumping every date to today teaches crawlers to ignore your dates.

You link your sitemap from robots.txt with a single line so crawlers find it automatically:

What Does llms.txt Actually Do?

Llms.txt is the new one, and the most misunderstood. It is a Markdown file at your site root that gives AI assistants a curated map of your best content — a short description of your site, then links to your most important pages, optionally with one-line summaries. The idea, from Jeremy Howard's llms.txt proposal, is that LLMs have limited context windows and benefit from a clean, link-rich summary rather than crawling and parsing your entire navigation, ads, and scripts.

Two honest caveats. First, it is a proposed convention, not an adopted standard — adoption by AI engines is uneven and unverified. Google has publicly said it does not use llms.txt as a ranking input. Second, the upside is unproven; our own 60-day study found mixed results. The downside, though, is near zero: a small, well-formed file costs an hour to write and cannot hurt you. Treat it as a low-cost bet, not a magic fix.

What Is the Minimal Correct Setup?

Here is the order to do this in, because sequence matters: get access right first, then discovery, then curation.

Before you call it done, run this quick check against the minimal correct setup:

The Misconfigurations We See Most

Three mistakes show up again and again in audits of Indian B2B sites.

The first is the staging block that shipped to production. A Disallow: / in robots.txt is correct on a staging server and catastrophic on the live site. It blocks everything from everyone. We have found live sites invisible to Google for weeks because a deploy carried the staging robots.txt across.

The second is treating llms.txt as a robots replacement. Llms.txt does not block anything and does not control access. If you want to keep a path out of AI training, that is a robots.txt job, not an llms.txt one. The two files do not overlap.

The third is the stale, dishonest sitemap — either listing dead URLs that 404, or stamping every page with today's date. Both teach crawlers to trust your sitemap less. A sitemap is only useful if it is accurate.

A Real Example: A SaaS Vendor in Bengaluru

A Bengaluru B2B SaaS vendor came to us frustrated that Perplexity never cited their product, despite strong organic rankings. Khushi, who leads our GEO audits, started where it matters: robots.txt. It carried a broad Disallow on their entire /docs path — added years earlier to keep docs out of Google, long since irrelevant — which also blocked PerplexityBot from the exact pages buyers asked about. We narrowed the rule, confirmed the sitemap listed the doc pages with honest dates, and added a one-page llms.txt linking the top product and comparison pages.

The fix was not the llms.txt; it was unblocking the docs. Within a few weeks, Perplexity began surfacing their documentation in answers to product questions. The lesson we keep relearning, and apply across our digital marketing engagements and on client sites like ChipMaker Hub: AI visibility usually fails at access first, not at curation. Check robots.txt before you write anything new.

Related Reading

For more on how AI engines weigh sources, our in-house English-learning app TalkDrill is a working example of an entity we maintain a clean content map for across these same files.

Frequently Asked Questions

Do I need all three files?

You need robots.txt and a sitemap — both are standard, well-supported, and affect how search and AI crawlers find and read your site. Llms.txt is optional: a low-cost, unproven bet on AI curation. Get the first two right before adding the third.

Does llms.txt help my Google rankings?

No. Google has publicly stated it does not use llms.txt as a ranking input. The file targets AI assistants that may use it to summarise your content more accurately, not Google Search ranking. Treat any ranking claim about llms.txt with caution.

Will blocking GPTBot remove me from ChatGPT answers?

Blocking GPTBot opts you out of OpenAI's training crawl. To stay in ChatGPT's live search citations, the relevant agent is OAI-SearchBot — block that and you drop out of cited answers. They are separate user-agents, so block or allow each deliberately.

What is the difference between robots.txt and llms.txt?

Robots.txt controls access — which paths a crawler may fetch — and is honoured by search and AI crawlers alike. Llms.txt controls nothing; it is a curated Markdown map suggesting your best content to AI assistants. One is a gate, the other is a recommendation.

Where do these files go on my site?

All three live at the site root: yoursite.com/robots.txt, yoursite.com/sitemap.xml, and yoursite.com/llms.txt. The sitemap is also linked from robots.txt with a Sitemap: line so crawlers discover it automatically.

How often should I update these files?

Robots.txt changes only when your access rules change, but re-check it after every site migration. Your sitemap should update automatically as pages change, with honest lastmod dates. Refresh llms.txt when your most important pages change — a few times a year is usually enough.

Can you audit my three-file setup?

Yes. We check robots.txt for accidental blocks, validate your sitemap and its dates, and write or fix your llms.txt as one coherent set. Most issues we find are over-broad robots.txt blocks, not missing llms.txt files.