Cloudflare has reversed its AI crawler policy from an optional block to a default block, fundamentally changing how AI bots can access content on the websites it protects. Previously, website owners had to opt in to block AI crawlers, but now, all new Cloudflare sites automatically block AI bots unless explicit permission is granted by the site owner. This makes Cloudflare the first major internet infrastructure provider to enforce a permission-based, opt-in model for AI content access.
Key aspects of the new policy
By default, AI crawlers—including those from major companies like Google, Meta, OpenAI, and Anthropic—are blocked from scraping content on Cloudflare-powered sites unless the site owner explicitly opts in to allow them. Website owners can now decide on a granular level which AI crawlers to allow, block, or charge for access. Cloudflare provides a dashboard for managing these settings and for monitoring crawler activity.
Cloudflare has introduced a “Pay Per Crawl” system, allowing site owners to set a micropayment rate for each crawl attempt by AI bots. This means content creators can demand compensation from AI companies that want to use their content for training or inference.
AI companies must now obtain explicit permission from website owners before scraping content. The system requires agreement from all parties: the website must opt in, and the AI company must disclose the intended use of the data (training, inference, or search), so site owners can make informed decisions.
The move is backed by major publishers and platforms, including Condé Nast, The Associated Press, Reddit, and Pinterest, who have committed to the default block in pursuit of a more sustainable and equitable digital ecosystem.
Rationale
Cloudflare’s decision is a response to concerns from content creators and publishers that AI companies have been exploiting their work without consent or compensation. Unlike traditional search engines, which drive traffic and ad revenue back to the original sources, AI bots often use content without attribution or benefit to the creators, leading to a potential erosion of incentives for producing original material.
