Common Crawl analysed more than 500,000 llms.txt files from its July 2026 crawl archive, and the picture is not flattering. Most sites have a template file generated by a plugin with nothing meaningful in it, and a subset are using it as a robots.txt substitute, writing crawler rules the format cannot enforce.
Key facts
- In effect: Not a policy change. This is a data analysis based on the July 2026 crawl archive
- Who it affects: Site owners whose SEO plugin generated an llms.txt they believe blocks AI crawlers
- What to do: Look at your own file and see if it is an empty template. To actually control crawlers, edit robots.txt
What happened
The numbers tell the story. 68% of files come from a plugin or template, 22.56% contain no links whatsoever, and only 49% have a complete structure. Wix accounts for 41% of the entire sample. The 73,000 files produced by the All in One SEO plugin never include summaries and carry a median of 138 links.
Two other figures matter more. At 136,578 URLs, the llms.txt path actually returns robots.txt content, roughly 10% of the 1,287,207 successful responses. Separately, 1,570 files name specific crawlers, and 32 appear to refuse Common Crawl's own CCBot. Checked one by one, not a single one of them actually blocks CCBot in robots.txt. The report includes one telling case: a site whose llms.txt declares it has blocked CCBot since June 2026, while its robots.txt permits that crawler through a wildcard rule. The two files have simply drifted apart.
Common Crawl puts it bluntly in the report: the file "grants nothing and blocks nothing, and no crawler is obliged to read it." The analysis also surfaced a small number of prompt injection attempts, four confirmed as genuine, including one instructing models to ignore prior instructions and fetch a different file.
What this means for independent store owners
The important part first. If you believe the llms.txt in your site root is keeping unwanted AI crawlers out, it is not. The format carries no authority and crawlers read it only if they feel like it. robots.txt is the file that gets obeyed, and where the two disagree, robots.txt wins. If you want control over who fetches your content, edit robots.txt and stop writing rules into llms.txt.
Now the other half. Most llms.txt files are empty shells a plugin created during setup, with no curated links and no summaries, which does nothing for AI citation. The format exists to point a model at the handful of pages on your site worth reading. Making it useful means writing it by hand: pick the dozen or so pages you most want cited, and give each one a line explaining what it covers. Leaving the default with 138 links and zero summaries is the same as not having one.
For perspective, a June report based on Ahrefs data found 97% of llms.txt files had never been requested at all. So keep this low on your list and do not let it displace real work. Ten minutes to confirm your file is not misleading you is enough.