Major Publishers Block Internet Archive Crawler

Major news publishers, including The New York Times, USA Today and The Washington Post, have blocked the Internet Archive's web crawler in recent months, limiting the organization's ability to preserve snapshots of their online content. The restrictions, affecting at least 23 prominent outlets according to an analysis by the artificial intelligence detection firm Originality AI, stem from concerns that archived material could be used by AI companies to train models without permission, potentially violating copyright laws.

The Wayback Machine, a project of the nonprofit Internet Archive, has archived more than a trillion web pages since its inception 30 years ago. It captures historical versions of websites, allowing users to track changes, verify information and access content from defunct sites. By blocking the ia_archiverbot crawler, publishers prevent new captures, though previously archived pages remain accessible. Social media platform Reddit has also imposed a full block, while The Guardian has adopted a partial approach, excluding its content from the archive's public interface and application programming interface without halting the crawler itself.