AI Firms Accused of Ignoring Robots.txt and Scraping Content

AI Companies Bypass Rules to Scrape Content, Spark Disputes with Publishers

Several AI companies are ignoring the Robots Exclusion Protocol (robots.txt) to grab content from websites without permission, says TollBit, a content licensing startup. This has caused conflicts between these AI firms and publishers. For instance, Forbes accused the AI company Perplexity of copying its content.