llms.txt and AI crawler management: an access strategy
A framework for deciding what to open to AI crawlers: separating training, search and user bots in robots.txt, where llms.txt fits, and log checks.
By Roozbeh Nazari · CEO
A significant share of the bots visiting your site today are no longer search engines. There are three distinct bot families: those collecting content for model training, those building an index for AI search results, and those fetching a page on demand when a user asks a question. All of them read the same robots.txt, but what each one gives your site and takes from it is different. This article explains how to decide what to open to which bot, and where llms.txt stands in that decision.
Access strategy is the first step of any AI search visibility work; without knowing which bot cannot see your page, you cannot interpret which answer you are missing from. The content side of visibility, how a passage becomes citable, was covered in our passage citability article.
Three bot families, three separate decisions
The major providers' documentation draws this distinction itself. OpenAI defines three main user agents: GPTBot crawls for model training, OAI-SearchBot builds the index for ChatGPT's search feature, and ChatGPT-User fetches a page at a user's request. Anthropic documents the same trio as ClaudeBot, Claude-SearchBot and Claude-User. On Google's side, Google-Extended is a standalone product token that manages whether content crawled from your site may be used to train Gemini models; Google's documentation states explicitly that it does not affect Search ranking.
The practical consequence is this: "should we block AI bots" is not one question but three. Closing to training bots while staying open to search bots is possible, and the documentation supports it explicitly; OpenAI writes that each setting works independently and that a site can allow OAI-SearchBot while disallowing GPTBot. For user-triggered bots there is a separate note: because they are not automated crawling, robots.txt rules may not apply in every case.
Decision matrix: what to open to which bot
The decision should be made by page type. The framework below summarises the practice we apply as an agency on clinic and B2B sites; it will not give the same result on every site, but it asks the questions in the right order.
- Search bots (OAI-SearchBot, Claude-SearchBot and the like): open on every page you want to be seen. Closing them means withdrawing from AI search results; OpenAI says so directly.
- Training bots (GPTBot, ClaudeBot, Google-Extended): according to the site's business model. Leaving them open on blog and guide content may contribute indirectly to visibility; closing them on original data, research and paid content is a defensible choice.
- User-triggered bots (ChatGPT-User, Claude-User): open. The moment this bot arrives is the moment a real user is about to see your page inside an answer.
The most common mistake we see on clinic sites in Türkiye is the "we are closed to AI" decision taken in a single line, blocking the search bots along with everything else. A few months later the site owners ask "why doesn't ChatGPT recommend us"; the answer is in their own robots.txt.
robots.txt: what it really does and does not do
Google's documentation states the limit of robots.txt plainly: the file tells crawlers which URLs they can access and is mainly used to avoid overloading the site with requests; it is not a mechanism for keeping a page out of Google. To keep a page out of the index you need noindex or password protection. The same logic applies to AI bots: robots.txt is a request, not a technical wall. The major providers with documentation state that they honour the request; for undocumented bots the only guarantee is server-side access control.
In practice, writing the file in three blocks improves readability: search engine bots, AI search and user bots, AI training bots. Noting in a comment line why each path is open or closed in each block saves time for whoever opens the file six months later. OpenAI writes that robots.txt changes take roughly 24 hours to propagate in its systems; making the change and looking for results in the log the same day is misleading.
llms.txt: what it is, and what it is not
llms.txt is a proposed file placed at the site root that describes, in Markdown, what the site is about and where the important resources are, for language models. The proposal's own text defines the format: an H1 title, a summary in a blockquote, then annotated link lists under H2 headings. The aim is for a model to understand the site without processing all of its HTML.
Honesty is needed here. llms.txt is not an access control mechanism; it blocks no bot and binds no bot. Google's crawler documentation does not list it as a supported mechanism, and OpenAI's and Anthropic's bot documentation rely on robots.txt. So llms.txt is not a replacement for robots.txt; at most it is an introductory file alongside it. Publishing one costs little, but expecting visibility from it is an unmeasured assumption.
Our practical advice: prepare an llms.txt, but let it contain the site's fifteen or twenty best pages, not the whole sitemap. The only way to learn who reads the file is to count requests to that path in the server log; track that count for three months after publishing and revisit the decision only then.
Log verification: behaviour, not the file
An access strategy is not something written in robots.txt; it is something seen in the log. The monthly job is to group requests in the server log by user agent name and ask three questions: is the bot you blocked still coming, is the bot you opened coming, and which pages is it fetching. If the answer to the first is "yes", either the rule is written wrong or the bot is arriving under an undocumented identity.
User agent names can be spoofed. That is why Google and the major providers publish the IP ranges of their bots; on suspicious traffic, verify the IP range rather than the name. If you are seeing an AI-related load problem on the site, the fix is usually not to block every bot but to separate the spoofed traffic and apply Crawl-delay or a server-side rate limit to the real ones.
We keep this verification as a fixed item in every technical SEO audit; the gap between the latest robots.txt and the behaviour in the log is the only measure of what the site is actually telling which bots.
Conclusion
AI crawler management is not a single blocking decision but three separate answers to three bot families. Staying open to search bots is the precondition of visibility, training bots are a business-model decision, and user bots are almost always open. robots.txt is where those decisions are written; log verification is the only proof that they are actually being applied. llms.txt is a small introductory file inside that picture; preparing one does no harm, but it is not the strategy itself.