Cloudflare Blocks AI Crawlers: New Default Setting
- Cloudflare, a major internet infrastructure provider, announced Tuesday it will now automatically block known AI web crawlers.
- The "Pay Per Crawl" program enables select publishers to set their own prices for AI scrapers to access their content.AI companies can then review the pricing and decide...
- Cloudflare has been developing tools to combat AI crawlers. In 2023, the company allowed websites to block crawlers respecting the robots.txt file.
Cloudflare is shifting its default setting to block AI crawlers, a move designed to protect original content and fairly compensate creators. Publishers now have the option to implement a “Pay Per Crawl” system, setting fees for AI access to their content. Cloudflare aims to empower content creators while allowing AI companies to innovate responsibly; in 2023, they allowed websites to block crawlers respecting the “robots.txt” file. This shift reflects a growing concern about unauthorized content scraping and the need to ensure AI companies utilize quality content ethically. Supported by prominent platforms; it’s a core change supported by leaders in tech. News Directory 3 is watching closely. What steps will Cloudflare take next to balance content protection and AI advancements? Discover what’s next …
Cloudflare to Block AI Crawlers by Default, Offering ‘Pay Per Crawl’ option
Updated July 1, 2025
Cloudflare, a major internet infrastructure provider, announced Tuesday it will now automatically block known AI web crawlers. This move aims to prevent unauthorized content access and ensure fair compensation for creators. New domain owners will be asked if they wish to allow AI scrapers, and some publishers can implement a “Pay Per Crawl” system.
The ”Pay Per Crawl” program enables select publishers to set their own prices for AI scrapers to access their content.AI companies can then review the pricing and decide whether to pay the fee or forgo the content. Cloudflare intends to expand this program, ensuring “AI companies can use quality content the right way – with permission and compensation.”
Cloudflare has been developing tools to combat AI crawlers. In 2023, the company allowed websites to block crawlers respecting the robots.txt file. Last year, websites could block “all” AI bots, nonetheless of robots.txt compliance. Cloudflare identifies these scrapers using a list of known AI bots.In March, Cloudflare introduced an “AI Labyrinth” to deter unauthorized scraping.
Several prominent publishers and platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, support Cloudflare’s AI crawler restrictions. These companies are adapting to a landscape were AI chatbots increasingly influence how people find data. Cloudflare CEO Matthew Prince said last week at an Axios Live event that “People trust the AI more over the last six months, which means they’re not reading original content.”
Cloudflare is also collaborating with AI companies to verify their crawlers and clarify their purpose,such as training,inference,or search. Website owners can then use this information to decide which crawlers to permit.
“Original content is what makes the Internet one of the greatest inventions in the last century,and we have to come together to protect it,” Prince said. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”
What’s next
Cloudflare will continue refining its AI crawler detection and “Pay Per Crawl” system, aiming to balance content protection with AI innovation. The company plans to onboard more publishers and AI companies to the program in the coming months.