Bot access
Whether a non-browser client gets your page or a challenge screen.
What it is
Bot management at the CDN layer classifies traffic by user-agent, TLS fingerprint and behaviour, then serves a challenge to anything that looks automated.
Legitimate research agents fail those checks in exactly the same way scrapers do, because at the network level they are indistinguishable.
Why an agent cares
This is the hardest failure to see from the inside: your site works perfectly in every browser you own, while returning 403 to every agent that visits.
It is also rarely a decision anyone made. It is a default on a plan someone upgraded to two years ago.
How to fix it
Decide deliberately which automated clients you want, then allow them at the edge.
- 01Allow-list the published AI crawler user-agents in your WAF or bot-management rules.
- 02Serve them real HTML, not an interstitial.
- 03Rate-limit rather than block, so a burst degrades instead of failing.
- 04Test with
curl -A "ClaudeBot" https://yoursite.com -o /dev/null -w "%{http_code}".
How we test it
We request your homepage twice — once identifying honestly as AgentReadyBot, once with a browser user-agent — and compare. A difference is the finding. We never impersonate another vendor's crawler.