Cloudflare experiment will block AI bot scrapers unless they pay a fee

Cloudflare is implementing new controls for AI bot crawlers, allowing new domain customers to block them by default, with major publishers already opting in. They've also launched a "pay per crawl" beta, enabling websites to charge AI crawlers for content access. These measures aim to protect publishers' revenue and content while supporting AI innovation. Cloudflare's CEO stresses the importance of safeguarding original content creators and ensuring a sustainable internet model. The company anticipates these tools will evolve and supports diverse marketplaces for AI crawler content access.