Publishers Blocking AI Crawlers Are Reshaping the Economics of Training Data

Major news publishers are blocking AI crawlers such as OpenAI’s GPTBot via robots.txt and similar controls.

The New York Times, CNN, ABC, The Guardian, BBC and others are among those restricting access; Reuters Institute said about 48% of leading news sites had blocked OpenAI crawlers by late 2023.

This reduces the pool of freely crawlable journalism for model training and pushes developers toward paid licensing, alternative sources, or retrieval-based systems.

It also gives publishers more control over how their reporting is used in AI products and shifts the economics of training data.