The countdown
Proxies are only half of a scraper that works. These ten habits, counted down from useful to essential, make the other half.
10. Log every response code
You cannot fix what you cannot see. Count 200s, 403s, 407s and 429s per proxy and per target, and you will spot problems before they become bans.
9. Set sensible timeouts and retries
10 to 30 seconds for residential, shorter for datacenter, and retry on a new IP rather than hammering the same one. See proxy timeouts.
8. Block images, fonts and video
If you only need text, do not download the rest. With per-GB pricing this can cut costs dramatically. See per-GB pricing.
7. Match the proxy's location to the target
A US retailer is fastest, and least suspicious, from a US IP. See geo-targeting.
Proxy tip: If you want to try these ideas on a real network, Geonode sells pay-as-you-go residential bandwidth that doesn't expire, so a small balance is enough to experiment with rotation, sticky sessions and geo-targeting. See our Geonode proxies guide for how its products fit together.
6. Use sticky sessions for multi-step flows
Pagination behind a search, logins and carts need one IP for the whole journey. See sticky sessions.
5. Keep headers, user agent and TLS consistent
A Chrome user agent with a Python TLS fingerprint is a giveaway. Real browsers or careful HTTP clients avoid it. See TLS fingerprints.
4. Back off on 429 and honour Retry-After
A 429 is a polite warning. Ignoring it turns into a ban. See what 429 means.
3. Cache what you already have
Do not re-download pages that have not changed. It is cheaper, faster and kinder to the site.
Budget tip: If you want a cheap proxy to try this with, Webshare has a free plan with 10 datacenter proxies and 1 GB of bandwidth a month, no card required. Paid lists let you pick proxy count, bandwidth and location. See Webshare for web scraping.
2. Pick the cheapest proxy type that works
Try datacenter first. Move to ISP, residential or mobile only for the targets that need it. See residential vs datacenter.
1. Read robots.txt and the site's terms
The most important tip costs nothing. Respect what the site asks, avoid personal data, and keep your load light. See ethical web scraping.
Residential tip: Rayobyte is worth testing when you need precise targeting: its residential proxies can be pinned to a country, state, city, ZIP code or ASN, with sticky sessions from 1 to 120 minutes. More in Rayobyte for web scraping.
Learn more
- Web scraping proxies
- How to avoid IP bans when scraping
- Proxy rotation strategies
- What is a honeypot trap?
- RFC 9309, robots.txt
- MDN on Retry-After
- Scrapy downloader middleware
- The EU's GDPR, for anything involving personal data
Sources and further reading
Independent references that back up the points above, from standards bodies, governments, universities and non-profits. They open in a new tab.
- RFC 9309: Robots Exclusion Protocol (IETF / RFC Editor): The official standard for robots.txt, which polite scrapers and crawlers read first.
- Retry-After header (MDN Web Docs (Mozilla)): The header that tells clients how long to wait before retrying.
- Downloader Middleware (Scrapy documentation): Scrapy's documentation for the middleware where proxy rotation lives.
- Regulation (EU) 2016/679 (GDPR) (EUR-Lex, European Union): The full text of the EU's General Data Protection Regulation.
- RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3 (IETF / RFC Editor): The TLS 1.3 standard that keeps HTTPS traffic encrypted even when it passes through a proxy.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.