Major AI Firms Adopt Unified Opt‑Out Standard for Web Scraping

OpenAI, Google, Microsoft and Anthropic announced they will adopt a new IETF‑backed standard that lets publishers opt out of AI scraping. The mechanism, a “policy token” inserted into a site’s robots.txt file, signals to all major AI systems that the content should not be used for training. Kevin Kelley, Anthropic’s standards lead, developed the token as part of the Internet Engineering Task Force. By placing a single token, site owners can set their AI‑training preferences once and have it respected by every participating company, rather than configuring each crawler separately. The move follows earlier reports that publishers have struggled to control how their content is harvested for machine‑learning models.