Cloudflare and OpenAI pilot surfaces hundreds of millions of new pages for ChatGPT Search
Cloudflare and OpenAI say a research pilot has helped OpenAI's search crawler discover and index hundreds of millions of pages it had not found before. The trial ran across a cohort covering roughly 1% of Cloudflare's network, with the two companies publishing early results on 15 October, about three months after the pilot was announced on 8 July.
The signals Cloudflare supplies tell OpenAI's crawler how recently a page changed, what kind of traffic a site receives and whether a page has actually been updated. The companies point to pages such as a washing machine user guide, current passport requirements and a restaurant's menu and hours, noting these can matter even when few other sites link to them. The same signals are meant to cut wasted requests, including repeat attempts at files that do not exist and addresses that redirect to a homepage or duplicate other content.
Neither announcement says how often the newly found pages were cited in ChatGPT answers or how much traffic they sent, and OpenAI ranks indexed pages independently, so inclusion is no guarantee of a citation. The companies are also testing signals that flag individual harmful pages, such as a fake login form planted on one page of a legitimate site, rather than penalising the whole domain.
For site owners, the practical lever is crawler access. Blocking OAI-SearchBot keeps a site out of ChatGPT's search results, while allowing it and blocking GPTBot keeps pages eligible for search without consenting to training use. Cloudflare's AI Crawl Control, available on all plans, lists which AI services are hitting a site and lets the owner allow or block each one. Cloudflare has said it intends to widen access to the network signals later in 2026, without giving a date.
Reported by Search Engine Journal ↗
Topics: ChatGPT, Citations, Search policy