A 403, a CAPTCHA wall, or an empty response. Before you swap libraries or buy more proxies, diagnose the block — then apply the one fix that actually matches it.
Run one test: retry the request from a clean 4G/5G mobile IP. If it works, you had an IP-reputation problem — keep routing through mobile proxies. If you are still blocked, it is a fingerprint problem (fix your TLS/JA4 and browser profile) or a policy problem (the site wants you to identify via Web Bot Auth or pay per crawl). Diagnose first; don't guess.
Route the exact request through a 4G/5G mobile proxy, nothing else changed.
Drive a genuine browser (e.g. a stealth Chromium/Firefox build) over the mobile proxy instead of a raw HTTP client.
A real browser on a clean IP is being refused — the site is choosing to block unidentified automation.
| Symptom | Likely cause | Fix |
|---|---|---|
| Works on laptop, blocked on server | Datacenter IP reputation | Route through mobile proxy |
| Instant 403 before any JS runs | TLS / JA4 fingerprint mismatch | Use a real browser client + mobile IP |
| CAPTCHA / challenge loop | Low IP trust + headless signals | Mobile IP + genuine browser, slow down |
| Refused even from a real device | Site policy (no anon automation) | Sign (Web Bot Auth) or pay per crawl |
Three of four common blocks are solved at the IP/fingerprint layer — which is why a real mobile-proxy layer resolves the majority of "my agent got blocked" tickets. See the full model in how AI agents access the web in 2026.
In 2026, agents are usually blocked for one of three reasons: the IP is a known datacenter or flagged residential address (reputation), the client fingerprint does not match a real browser (JA4/TLS), or the site simply does not permit unidentified automation and wants you to identify or pay. Diagnose which signal tripped before changing anything: if you get in from a clean mobile IP, it was reputation; if you are blocked even from a real device IP, it is fingerprint or policy.
Sign your requests only on sites that support Web Bot Auth and allowlist you — there, identity is the cleanest access. Everywhere else (the default), signing just makes you easy to block, so route through real 4G/5G mobile proxies and behave like an ordinary user. If a publisher sells access through pay-per-crawl and the content is worth it, paying is the third option.
That is the classic IP-reputation signature. Your laptop is on a residential or mobile connection with good reputation; your server is on a datacenter ASN that anti-bot systems block on sight. The fix is to give the server traffic a trustworthy IP by routing it through a mobile proxy.
Real 4G/5G mobile proxies across 17+ countries — $4/GB, free endpoints, free rotation, MCP + x402 for agents.