Blocks are a signal, not a mystery
When a website blocks your scraper, it's responding to signals: too many requests from one place, an IP type associated with bots, a client that doesn't look like a browser, or behaviour no human would produce. Fix the signals and the blocks go away. This guide lists the techniques that matter most, roughly in order of impact. For the bigger picture, see web scraping proxies.
Proxy-level techniques
1. Use the right IP type for the target
Datacenter IPs are fine for many sites and instantly suspicious on others. Use residential proxies for protected targets and datacenter proxies where they work. See residential vs datacenter proxies.
2. Rotate IPs appropriately
Spread requests across enough IPs to keep each well under the site's limits. Rotate per request for independent pages; use sticky sessions for flows. See what is proxy rotation.
3. Use clean IPs
Burned IPs and subnets get blocked immediately. Choose providers with good reputation management and retire IPs that fail. See IP reputation and proxies.
4. Match location to the site
A local site serving local users may treat foreign traffic with suspicion. Use geo-targeted proxies.
Rate and timing techniques
5. Throttle per domain, not just per IP
Websites watch their total load. A sudden 50x spike in traffic is noticed even when spread across 10,000 IPs. Cap concurrency per domain. See rate limiting and proxies.
6. Add jitter
Regular, machine-precise intervals are a giveaway. Randomise delays.
7. Back off on 429 and 503
Respect Retry-After headers. Exponential backoff with jitter is the standard approach.
8. Scrape at sensible times
Off-peak hours for the target often mean more tolerance and less impact on real users.
Client techniques
9. Send realistic headers
Use a current browser user agent and the headers a real browser sends (Accept, Accept-Language, Accept-Encoding, sec-ch-ua hints for Chromium). Keep them consistent within a session.
10. Mind your TLS fingerprint
HTTP libraries have TLS signatures that differ from browsers. Some bot protection blocks on this alone. Use real browsers or TLS-impersonating clients for such sites. See browser fingerprinting and proxies.
11. Handle cookies like a browser
Accept and return cookies within a session. Some sites set a cookie on the first page and expect it on the next.
12. Use headless browsers only when needed, and harden them
Browsers pass JavaScript checks but leak automation signals. See Puppeteer stealth and proxies.
Proxy tip: If you need a pool to run this against, Geonode works with every tool in this guide: rotate per request for stateless pages, or hold a sticky session for logins and pagination. See Geonode for web scraping for working code.
Detection and recovery techniques
13. Detect soft blocks
Many blocks return HTTP 200 with a CAPTCHA, an empty template or decoy data. Validate content, not status codes.
def is_soft_block(html):
signals = ["captcha", "verify you are human", "access denied", "unusual traffic"]
return any(s in html.lower() for s in signals) or len(html) < 200014. Retire bad proxies quickly
Track success per IP or session; stop using ones that fail repeatedly.
15. Escalate rather than hammer
If datacenter fails, try residential; if residential fails, try an unblocker. Don't retry the same failing setup endlessly. See proxy rotation strategies and web unblocker proxies.
What not to do
- Ignore robots.txt and terms entirely without considering the consequences. See ethical web scraping.
- Scrape logged-in pages with rotating IPs.
- Run thousands of concurrent requests against a small site.
A worked troubleshooting example
Problem: a scraper on 50 dedicated datacenter proxies gets 403s on 60% of requests to a retailer.
- Check headers: default library user agent. Fix to a realistic browser header set. Blocks drop to 45%.
- Check TLS fingerprint: switch to a browser-impersonating client. Blocks drop to 25%.
- Check rate: 20 concurrent requests to the domain. Reduce to 5 with jitter. Blocks drop to 15%.
- Check IP type: move this target to rotating residential. Blocks drop to 2%.
Without steps 1 to 3, residential proxies alone might have reduced blocks only modestly, at a much higher cost. See residential vs datacenter proxies.
Building a simple block dashboard
Bans rarely arrive all at once. They creep in as a rising share of challenges and 403s, and a basic dashboard catches the trend while it's still cheap to fix. Track, per target and per hour:
| Metric | Why it matters | Alert when |
|---|---|---|
| Success rate (validated content) | The number that pays the bills | Drops more than 10 points against last week |
| 403 and block-page rate | IP or fingerprint flagged | Doubles within a day |
| 429 rate | You're going too fast | Any sustained rise |
| Challenge rate | Risk score rising | Climbs for three hours in a row |
| Median and p95 latency | Slow exits or throttling | p95 doubles |
| Bytes per successful page | Cost creep from retries or heavier pages | Rises 20% |
You don't need special tools. Write one line per request (time, target, status, validated or not, bytes, latency, proxy or session) to a file or table, and chart it with whatever you already use.
When an alert fires, look for what changed: your code, your rate, your proxy plan, or the target. The worked example above shows how to work through it. For deeper causes, see CAPTCHAs and proxies and rate limiting and proxies. If one site keeps tipping over, web unblocker proxies may be cheaper than fighting it yourself.
FAQ
Will rotating proxies alone stop IP bans?
They help a lot with IP-based limits, but not with fingerprint or behaviour-based detection.
Why am I blocked even on my first request?
Your IP range, TLS fingerprint or headers are flagged before behaviour even comes into play.
How long do IP bans last?
From minutes to permanent, depending on the site. Rotating to fresh IPs is usually faster than waiting.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.