If your conversion rate has drifted downward while sales held steady, or your bounce rate crept up without any obvious cause, there is a decent chance the numbers changed before the business did. AI crawlers, retrieval bots, and shopping agents now generate a meaningful share of requests to a typical small business site, and most analytics setups quietly count a portion of them as people.
A metric that mixes humans and agents is not a slightly noisier metric. It is a different metric, and every decision you make from it is being made about a population that does not exist.
Primo Collab
Why this breaks your numbers specifically
Agents behave in ways that are individually harmless and collectively distorting:
- They land on a page, extract what they need, and leave. That is a bounce, and a session duration near zero.
- They do not convert, because they were never buying. That is a denominator inflated with traffic that could never have converted.
- They hit product and pricing pages disproportionately, because that is where the structured data lives. Your highest-intent pages look like they are suddenly performing worse.
- They often do not execute JavaScript, so some are invisible to your analytics while still consuming server resources and appearing in server logs.
The net effect is a set of metrics that look like a gradual decline in site performance. Nothing declined. The population being measured changed.
Separating the two
There is no single switch for this, but there is a workable sequence.
Start with server logs, not analytics
Your analytics tool only sees what it sees. Server logs see everything, including agents that never run JavaScript. This is the only place you get a true picture of total request volume and who is generating it.
Identify by user agent, then verify
Most legitimate crawlers identify themselves in the user agent string. That gets you most of the way. It is worth verifying the major ones by reverse DNS lookup rather than trusting the string alone, since user agents are trivially spoofed.
Build a segment rather than deleting the data
The instinct is to filter agents out and forget them. That is a mistake, because agent traffic is becoming a channel worth understanding rather than noise worth suppressing. Create a segment so you can look at each population separately and at the total when you need it.
Fix your reporting baselines
Once you can separate the two, your historical comparisons are broken, because the past numbers are still mixed. Note the date you made the change and treat comparisons across it carefully for a couple of quarters.
What to actually watch, per population
| Metric | For humans | For agents |
|---|---|---|
| Conversion rate | The real number, finally | Not meaningful yet |
| Bounce rate | Genuine engagement signal | Expected to be near total |
| Pages per session | Navigation quality | Which data they are collecting |
| Top pages | What people want | What agents find machine-readable |
| Volume trend | Demand | Growing interest from AI surfaces |
| Errors and timeouts | UX problem | Whether agents can complete a request at all |
That last row deserves attention. If agents are hitting errors on your site, you never hear about it. There is no frustrated customer emailing you. There is just a system that stops trying.
What agent traffic can tell you that humans cannot
Once separated, agent traffic becomes a genuinely useful signal:
- Which pages agents fetch most tells you which of your content is machine-legible. It is often not the pages you would guess.
- Which pages they never touch points at content that is structurally invisible, usually because it depends on JavaScript or lacks structured data.
- Growth in agent volume is a leading indicator of AI surface visibility, ahead of any traffic showing up in your reports.
- Failed requests show you where an agent tried to do something and could not.
The practical minimum
If you do nothing else this month:
- Look at your server logs for one week and find out what percentage of requests are not human.
- Create one segment separating known crawlers and agents from everything else.
- Re-read your conversion rate with that segment excluded and see whether the number you have been worrying about was ever real.
The mistake to avoid
Making business decisions from blended numbers. Cutting a page that “does not convert” when most of its traffic was never capable of converting, or concluding your site got slower when what actually changed was who was requesting it. The fix is not sophisticated. It is knowing which population you are looking at before you draw a conclusion from it.