Google's AI crawler policy and publisher control debate
⚖️ Cloudflare welcomes the UK CMA’s consultation on proposed conduct requirements for Google but argues the measures do not go far enough to protect publishers and competition. Cloudflare’s analysis shows Googlebot accesses substantially more unique pages than other AI crawlers, giving Google an entrenched advantage that can undercut publisher revenue. The company urges mandatory crawler separation so sites can permit search indexing while blocking use of content for generative AI, restoring publisher choice and enabling fairer market competition.
