The rise of gray bots reshaping the digital economy

A new category of bots, dubbed "gray bots," is quietly becoming one of the most significant forces reshaping the web. These automated programs—including AI agents, scrapers, and crawlers—don’t fit neatly into traditional categories of "good" or "bad" bots. Instead, they present a paradox: fueling innovation while straining digital infrastructures and distorting analytics.

What Are Gray Bots?

Gray bots are automated programs that power AI search engines like ChatGPT and Perplexity, collect data for large language models, and enable emerging use cases like automated browsing and shopping. However, they also scrape content, bypass ads, inflate traffic metrics, and consume resources without providing value in return. Recent cybersecurity reports show some websites are seeing millions of scraper bot requests monthly, largely tied to generative AI activity.

Risks for Publishers and E-Commerce

Publishers and e-commerce businesses face growing risks from gray bots. These bots can:

• Scrape intellectual property and pricing data
• Access gated content without permission, undermining paywalls and affiliate models
• Mimic human behavior, inflating engagement metrics and triggering ad calls without delivering value

For example, the Wikimedia Foundation disclosed that bots and AI scrapers drove a 50% rise in infrastructure costs since January 2024.

Scale of the Problem

Last year, AI scraper bots accounted for 16% of general invalid traffic (GIVT), according to DoubleVerify. In 2024, GIVT nearly doubled, with four-quarter volume surpassing 2 billion ad requests for the first time. HUMAN Security reported blocking over 215 billion scraping attempts in 2024, with the largest growth in technology, SaaS, and services (nearly 500% year-over-year).

Where Are the Bots Coming From?

Major AI players like OpenAI, Perplexity, Google, and TikTok are driving gray bot traffic. For instance, chatbots like ChatGPT and Claude deploy gray bots to retrieve content, creating issues for ad-supported websites. Some publishers, like Reddit, have negotiated paid access agreements, while others block AI crawlers via robots.txt.

Detection and Solutions

Gray bots are increasingly difficult to detect due to their rapid growth and ability to mimic human behavior. Some evade detection by disguising as regular browsers or ignoring robots.txt rules. Solutions include:

• The EU’s legal framework (AI Act, GDPR, Digital Services Act, and Copyright Directive)
• New systems like Skyfire and Cequence’s "Know Your Agent" framework, which aims to monetize bot traffic by verifying identities and enabling site owners to charge for access

As Amir Sarhangi, CEO of Skyfire Systems, noted, "The site’s already being accessed today. It’s already being scraped." The challenge now is managing this new wave of non-human traffic without stifling innovation.