llms.txt adoption grows 17.5x in 14 months, but no AI platform has confirmed using it
The number of domains publishing an llms.txt file has risen from 951 in July 2025 to 16,670 by September 2026, roughly 17.5x growth in 14 months. The file is a proposed standard that gives AI crawlers a curated Markdown list of a site's most important pages, so they spend their crawl budget on product and content pages rather than navigation and cookie banners.
No major AI platform has confirmed that it reads or relies on llms.txt. Google's John Mueller has said as much, and while Anthropic publishes a file on its own site, that is not evidence its crawler uses one. Google's Lighthouse auditing tool now includes an experimental check for the file, but Lighthouse audits and live crawling are separate systems.
Testing on Search Engine Land found no correlation between publishing an llms.txt file and improved performance in AI results, and server log analysis did not show AI crawlers fetching it. Google has also released the Open Knowledge Format, an early specification for packaging content for AI agents, though it faces inward at an organisation's own agents rather than outward at visiting crawlers.
For brands and agencies, the file is a low-cost experiment rather than a requirement. Content can still appear in AI answers without one, and the field remains uncrowded: 16,670 domains is a small fraction of active websites, so publishing now puts a site ahead of most competitors without any confirmed payoff.
Reported by Semrush Blog ↗
Topics: Citations, Google Search, Measurement