Cloudflare Introduces New Setting to Control AI Training

Cloudflare has announced a new setting called Disallow AI Training, which allows website owners to stay indexed for search while refusing to let mixed-use crawlers train on their content. This setting is available for Apple, Google, and Microsoft, which have committed to honoring it. Cloudflare has also introduced a new classification system for bots, which includes Search, Training, and Agent behaviors. The Disallow AI Training setting is a response to the tradeoff between allowing content to be used for AI training and risking losing discoverability in search. Cloudflare has been working with bot operators to establish a new standard for transparency and control, and has created an 'Accountable' designation for operators that meet certain requirements. The new setting is part of Cloudflare's Bot Management and AI Crawl Control system, and will become part of the recommended configuration for certain new domains.