A file that asks nicely is not a fix
Sometime last year llms.txt showed up in AEO advice threads as the thing you must ship. The pitch is clean: put a markdown file at your root, list your most important pages and a short description of what your site is, and AI systems will lean on it to understand and prioritize your content. It reads like robots.txt for the LLM era.
Here's the part nobody wants to say out loud. llms.txt is a proposed convention, not a standard anyone is obligated to follow. There is no enforcement, no spec that browsers or crawlers must honor, and no built-in confirmation that a given engine reads the file at all. robots.txt earned its authority over decades because the major crawlers agreed to respect it and search relevance depended on that trust. llms.txt has no such contract yet.
So most teams that added one didn't actually change anything. They shipped a file, checked a box, and assumed the assistants started prioritizing the right pages. Assuming is the whole problem. A suggestion only helps if someone on the other end chooses to act on it, and you have no idea whether they did until you look at your own logs.
robots.txt vs llms.txt: why one is load-bearing and one is a request
| Comparison category | Property | robots.txt | llms.txt |
|---|---|---|---|
| Status | De facto standard, honored for ~30 years | Proposed convention, informal | |
| Who commits to reading it | All major search and AI crawlers | No engine formally commits | |
| Enforcement | Ignoring it breaks crawler trust norms | None. Purely voluntary | |
| Confirmation you can check | Crawler behavior visibly respects rules | Only your server logs reveal reads | |
| Downside of skipping it | Real: unwanted crawling, wasted budget | Low: you lose an optional hint |
Where Crescive comes in: proof instead of hope
The setup is the easy 20 minutes. The verification is the part that separates a real improvement from a placebo, and it's the part almost nobody does because it requires reading crawler logs across a half-dozen different agents and knowing which ones matter.
Crescive's crawler analytics watches the AI agents that hit your site: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and the rest. After you ship an llms.txt, it tells you which agents actually requested the file, whether their fetch patterns shifted toward the URLs you flagged as important, and which engines ignored the whole thing. That turns a guess into evidence. If PerplexityBot starts pulling your prioritized pages and GPTBot doesn't budge, you know exactly where the file is earning its keep and where you still need on-page and citation work to move the needle.
This matters because the alternative is optimizing blind. You spend a quarter believing a file fixed your AI visibility while the engine that drives most of your assistant traffic never read a byte of it.
Ship an llms.txt and actually confirm it did something
- Write the file: create /llms.txt in markdown with a one-line site summary, then a linked, prioritized list of your highest-value pages (docs, product, pricing, definitive guides) with a short note on each.
- Keep it honest and small. List the pages you genuinely want cited, not your whole sitemap. A bloated file dilutes the signal.
- Deploy it to your root and confirm it returns a clean 200 at yoursite.com/llms.txt with the correct content type.
- Set a baseline. Record current AI crawler activity by agent before the file has time to matter, so you have a real before-and-after.
- Wait two weeks, then check your logs by agent: did any AI crawler request /llms.txt, and did fetches to your prioritized URLs rise for the agents that read it?
- Compare against the baseline. If a given engine shows no change, treat the file as inert for that engine and fix the underlying content and citations instead.
- Re-check monthly. Adoption of this convention is moving, so an engine that ignores it today may start reading it next quarter.
Treat it as one lever, not the lever
llms.txt is worth shipping. It's cheap, it can't hurt, and the engines that do read it get a cleaner map of your best content. Just don't mistake the file for the outcome. The outcome is measured in whether assistants describe and cite you correctly, and that depends far more on what's on your pages than on a hint file at your root.
Publish it, then verify it, then keep doing the harder work of making the actual content quotable. The teams that win at AEO are the ones who refuse to assume, and who go check the logs before they claim a fix.
Key takeaways
- llms.txt is a voluntary convention with no enforcement and no guarantee any engine reads it. Publishing one is a hint, not a fix.
- The only way to know it worked is to compare AI crawler behavior before and after you ship it, agent by agent.
- Crescive's crawler analytics shows which AI agents read your llms.txt and whether their fetch patterns actually shifted toward your prioritized pages.
- Keep the file small and honest, then treat on-page content and citations as the real driver of AI visibility.
FAQ
Does adding an llms.txt file guarantee AI assistants will prioritize my content?
No. llms.txt is a proposed, voluntarily-honored convention, not an enforced standard like robots.txt. No AI engine is obligated to read it, and none confirm that they did. The only way to know it changed anything is to check your server logs and see whether specific AI crawlers requested the file and shifted their fetches toward the pages you flagged.
How do I verify whether an AI crawler actually read my llms.txt?
Record a baseline of AI crawler activity by agent before you ship the file, then check your logs after about two weeks for requests to /llms.txt and any change in fetches to your prioritized URLs. Crescive's crawler analytics automates this by tracking agents like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended and showing which ones read the file and changed behavior versus which ignored it.