Crawled a lot, referred nothing
By Allan Leone on
Automated requests are now about 57 percent of web traffic, and the agents doing the clicking grew roughly 8,000 percent in a year. The number that should worry you is not the traffic. It is the referrals that did not come back.
Bots passed humans a while ago. The current measurements put automated requests at around 57 percent of all HTTP requests, and traffic from agents that actively click links and fill in forms has grown by something close to 8,000 percent in a year. I wrote about the crossover before. What changed this quarter is the shape of the exchange, not the volume.
DataDome logged 17.7 billion AI agent requests in the second quarter. Meta's crawlers accounted for 9.1 billion of them and sent almost nobody back. That is the part worth sitting with. The old bargain was that a crawler took your content and returned attention. One side of that is quietly being withdrawn.
Why this is a product problem, not an ops problem
The instinct is to treat it as infrastructure: rate limits, bot rules, a bigger cache. Those are worth doing and they do not address the actual change, which is that a growing share of your audience never arrives as a person.
If an agent reads your pricing page and reports back to its user, your pricing page is now a data source with an interface stapled to it. The comparison is happening off your property. Nothing you do with a hero animation participates in that decision.
What we changed
- Facts that used to live only in layout now also exist as structured data. Prices, availability, dates, contact routes.
- Anything decision-relevant is in the served HTML rather than assembled after a JavaScript render.
- Comparison-shaped information is written plainly instead of implied by design.
None of that is a concession. Writing down what is true in a form a machine can lift is the same discipline as writing a good spec, and it makes the human page better as a side effect, because vagueness has nowhere to hide.
The measurement problem
Your analytics were built for sessions. An agent that reads six pages in one second and produces one answer for one person shows up as inflated traffic and zero engagement, and both numbers are lies about what happened.
We stopped reading traffic as a proxy for interest. Inbound enquiries that mention something specific from the writing tell us more than any pageview chart currently can. It is a cruder instrument. It is measuring the right thing.
Tags: ai, product, business