"In the fourth quarter of 2025, TollBit estimates that an average of one out of every 31 visits to its customers’ websites was from an AI scraping bot. In the first quarter, that figure was only one out of every 200. The company says that in the fourth quarter, more than 13 percent of bot requests were bypassing robots.txt, a file that some websites use to indicate which pages bots are supposed to avoid. TollBit says the share of AI bots disregarding robots.txt increased 400 percent from the second quarter to the fourth quarter of last year.
TollBit also reported a 336 percent increase in the number of websites making attempts to block AI bots over the past year. Pangrahi says that scraping techniques are getting more sophisticated as sites try to assert control over how bots access their content. Some bots disguise themselves by making their traffic appear like it’s coming from a normal web browser or send requests designed to mimic how humans normally interact with websites. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic.
TollBit markets tools that website owners can use to charge AI scrapers for accessing their content. Other firms, including Cloudflare, offer similar tools. “Anyone who relies on human web traffic—starting with publishers, but basically everyone—is going to be impacted,” Pangrahi says. “There needs to be a faster way to have that machine-to-machine, programmatic exchange of value.”
WIRED attempted to contact 15 AI scraping companies cited in the TollBit report for comment. The majority did not respond or could not be reached. Several said that their AI systems aim to respect technical boundaries that websites put in place to limit scraping, but they noted such guardrails can often be complex and difficult to follow."
https://www.wired.com/story/ai-bots-are-now-a-signifigant-source-of-web-traffic/
