AI web crawlers are destroying websites in their never-ending content hunger
SMRTR summary
Thirty percent of global web traffic now comes from bots, with AI data fetchers leading the charge. Behind those numbers lies a silent crisis for website owners across the digital landscape.
"AI bot attacks – and let's face it, they are attacks – not so much," notes one affected site owner whose small tech page gets regularly knocked offline despite having protection against traditional denial-of-service attacks.
Unlike traditional crawlers that index content for search engines, AI bots from Meta, Google, and OpenAI hammer websites with traffic spikes reaching twenty times normal levels within minutes. These digital tsunamis, sometimes reaching 30 Terabits in a single surge, ignore standard crawling protocols while extracting massive amounts of content.
The consequences ripple through the internet ecosystem. Small businesses on shared servers experience slowdowns even when not directly targeted. Larger sites must invest in additional resources just to stay operational.
Most frustrating for content creators: these AI bots strip-mine websites without sending visitors back or generating revenue.
While some protection tools exist, the arms race between website owners and AI companies threatens to fragment the web behind paywalls and access restrictions, potentially ending the era of open internet access.
SMRTR provides this summary for quick context. The original article belongs to Hacker News.
Read the original article