Why Amazon is a hard target
Amazon is one of the most scraped sites on the web, and its defences reflect that. It quickly challenges datacenter IPs and fast, uniform traffic with CAPTCHAs or "robot check" pages. Results also vary by marketplace, delivery location and sometimes by session. Background in Amazon scraping proxies.
Which Webshare product
For product pages, search results and offers, use Webshare's rotating residential traffic. Datacenter IPs tend to be challenged quickly on Amazon. For seller account work, use the official seller tools and APIs rather than proxies, and if you must log in, use a dedicated static residential IP that matches the account's location.
Marketplace and location
- Marketplace: use IPs in the marketplace's country (US IPs for amazon.com, UK for amazon.co.uk, and so on).
- Delivery location: prices, delivery dates and some offers depend on the delivery ZIP or postcode, usually set in a cookie. Set it at the start of a session and keep the IP stable while you collect.
- ZIP-level IPs: Webshare's residential targeting supports ZIP codes, useful when you want IP location and delivery location to agree. See Webshare geo-targeting.
Pacing and sessions
- Rotate per product page for bulk collection.
- Use a session when paging through search results or reviews.
- Spread requests and keep per-IP rates low.
- Detect robot checks by content, not status code; Amazon often returns 200 with a challenge page.
What to extract
- ASIN, title, brand, price, list price, coupon, availability, seller, Buy Box owner, rating, review count and best sellers rank.
- Store raw HTML for a small sample, so you can re-parse when layouts change.
Reducing bandwidth
Amazon pages are large. Fetch HTML without rendering where possible, and if you use a browser, block images, video and fonts. This cuts residential cost substantially.
Budget tip: Before scaling, run a short pilot with Webshare on your real targets and record success rate, latency and bandwidth per result. Our our complete Webshare guide explains which Webshare product fits which job.
Official alternatives
If you're a seller or vendor, Amazon's own APIs provide much of your data without scraping. For catalog-wide or competitor data, proxy-based collection remains common; make sure your use complies with applicable law and the site's terms.
Comparing providers
Amazon collection is bandwidth-heavy, so per-GB price matters. Geonode lists lower per-GB residential rates and offers a Scraper API that handles rendering and retries; compare in Webshare vs Geonode for Amazon.
A practical collection plan
- Define scope: marketplaces, categories or ASIN lists.
- Discover ASINs from search, category and best seller pages, using sessions for pagination.
- Fetch product pages with rotation, one ASIN per request.
- Set delivery location per session where prices or availability by location matter.
- Validate every page for robot checks and missing fields.
- Schedule refreshes based on how fast each category changes.
Bandwidth estimate
| Job | Pages per month | Approx. MB per page (HTML) | GB per month (90% success) |
|---|---|---|---|
| 5,000 ASINs, daily | 150,000 | 0.8 | About 133 |
| 20,000 ASINs, weekly | 80,000 | 0.8 | About 71 |
| Best seller lists, 200 categories, daily | 6,000 | 0.6 | About 4 |
Size your residential plan from numbers like these, then adjust after a week of real data.
Proxy tip: For a second opinion on hard targets, run the same test through Geonode residential proxies, which support sticky sessions up to 24 hours and ASN targeting. We compare the two in Geonode for Amazon data.
Common Amazon parsing pitfalls
- Prices split across several elements (whole and fraction).
- Different layouts for books, groceries, devices and apparel.
- "Currently unavailable" and "see all buying options" pages without a main price.
- Localised number formats in non-US marketplaces.
Reducing robot checks
- Keep request rates low and spread across the day.
- Use realistic, consistent headers per session.
- Avoid hammering the same ASIN repeatedly.
- Prefer fewer, more useful requests over many speculative ones.
A real-world example
A brand selling on Amazon in the US and UK monitors 400 competitor ASINs in each marketplace daily for price, Buy Box owner and rating. They use Webshare rotating residential traffic in each marketplace's country, one request per ASIN, with HTML-only fetching and content checks for robot pages. At about 0.8 MB per page, the program uses roughly 21 GB a month. When robot checks rise above 10% on one day, the script automatically slows its rate for the next run.
For the bigger picture beyond these two providers, read our best proxies buyer's guide, and see cheap proxies for ways to cut cost per successful request. To understand the IP type behind this, read our residential proxies guide.
Key takeaways
- Amazon challenges datacenter IPs, so use rotating residential traffic for product and search pages.
- Use IPs in each marketplace's country, and set the delivery location within a sticky session.
- Detect robot checks by content, since they often return a 200 status.
- Amazon pages are heavy; fetch HTML only and estimate GB before sizing a plan.
- Sellers should use Amazon's official APIs for their own data.
FAQ
Can I scrape Amazon with Webshare datacenter proxies?
You'll hit robot checks quickly. Use residential traffic for Amazon.
How do I get prices for a specific ZIP code?
Set the delivery location in the session and use a residential IP in the same area, keeping the session sticky.
Why do I get CAPTCHAs even with residential proxies?
Rate, headers and fingerprints matter too. Slow down, use realistic clients and rotate IPs. See CAPTCHAs and proxies.
How many Amazon pages can I fetch per GB?
At about 0.8 MB per HTML page, roughly 1,100 to 1,200 pages per GB before retries.
Can I collect Amazon reviews?
Review pages can be collected with sessions for pagination; check terms and handle reviewer data carefully.
Do I need different proxies for each Amazon marketplace?
Use IPs in each marketplace's country for accurate prices and availability.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.