What e-commerce scraping covers
E-commerce data collection usually means one or more of:
- Catalog crawling: discovering every product URL in a store.
- Product details: titles, prices, variants, images, specs.
- Stock and availability: often by store or delivery area.
- Reviews and ratings.
- Seller and marketplace data.
Each has different traffic patterns, so the proxy setup differs too. Background in best proxies for e-commerce scraping.
Product choice by store
| Store type | Webshare product |
|---|---|
| Independent stores and many Shopify-style shops | Datacenter list |
| Large retailers with bot protection | Rotating residential |
| Marketplaces | Rotating residential, with careful pacing |
| Location-dependent stock or prices | Residential with geo-targeting |
Webshare's mix of cheap datacenter lists and residential traffic in one account makes this split easy. Test each store first; see testing Webshare proxies.
Catalog crawling
- Use sitemaps when available. They're the cheapest way to discover URLs.
- Crawl category pages with sessions. Pagination often relies on cookies; use a session per category so the IP stays the same. See Webshare sticky sessions.
- Deduplicate URLs with tracking parameters stripped.
Product pages
- Rotate per request for independent product pages.
- Look for JSON. Many stores embed product data as JSON-LD or call an internal API; extracting that is lighter and more stable than parsing HTML.
- Validate that price and title exist, so block pages don't pollute your data.
Stock by location
Stock is often tied to a selected store or delivery postcode, stored in a cookie. Set it explicitly at the start of a session and keep the IP stable while you check products. Use residential IPs in the same region where the store checks location. See Webshare geo-targeting.
Reviews
Review pages are paginated and sometimes loaded through APIs. Use sessions for pagination, respect rate limits, and be careful with personal data in reviewer names and profiles; collect only what you need.
Budget tip: You can test most of this guide for free: Webshare gives every new account 10 datacenter proxies and 1 GB of bandwidth a month without a credit card. Move to a paid plan once the pilot works. See Webshare for web scraping.
Handling bot protection
Large stores use commercial bot protection that looks at IP type, TLS fingerprint, headers and behaviour. Residential proxies handle the IP part; you still need realistic clients. Options:
- A headless browser with resource blocking. See Webshare with Playwright.
- HTTP clients with browser-like TLS fingerprints.
- Slower, steadier request rates.
See avoid IP bans when scraping.
Costs
Datacenter lists cover most of the long tail of smaller stores at a fixed monthly price. Residential cost depends on GB per product page; JSON extraction and HTML-only fetching keep it low. For big residential volumes, compare per-GB costs with Geonode; see Webshare vs Geonode for e-commerce.
Store-by-store routing table
Keep a small configuration that records how each store is handled:
| Store | Product | Mode | Rate | Notes |
|---|---|---|---|---|
| smallshop.example | Shared DC | Direct | 1 req / 3 s | Sitemap available |
| bigretail.example | Residential | Backbone, session per category | 1 req / 5 s per session | JSON-LD on product pages |
| marketplace.example | Residential | Backbone, rotate | Low | Validate prices carefully |
Reviewing it monthly keeps costs aligned with each store's defences.
Extracting from structured data
Many e-commerce pages embed structured data:
- JSON-LD
ProductandOfferobjects with name, price, currency and availability. - Open Graph tags with titles and images.
- Internal APIs used by the page to load prices and stock.
Extracting these is more reliable than parsing visual HTML and often lets you skip rendering, which keeps bandwidth low.
Proxy tip: If your residential usage goes up and down, compare Geonode: its per-GB residential bandwidth doesn't expire, so you only pay for what you use. See Geonode for price monitoring.
Variants and options
Products with sizes, colours or bundles often load variant prices and stock dynamically. Look for the variant API call, or iterate options in a browser session. Record each variant separately with its own SKU.
Handling store changes
Stores redesign, change platforms and move URLs. Monitor per-store record counts and parsing errors; a sudden drop usually means a layout change rather than a proxy problem. Keep parsers per store small and well tested so fixes are quick.
A real-world example
A dropshipping analytics tool tracks bestsellers across 1,500 small online stores and three large marketplaces. The small stores mostly expose product JSON, so a Webshare datacenter list fetches those cheaply every night. The marketplaces need residential traffic; the team collects only category bestseller pages there, not every product, which keeps residential usage to about 20 GB a month. Customers get daily trend data at a proxy cost of a few cents per store.
If you're still shortlisting, compare this setup with the options in our best proxies guide, and use our cheap proxies guide to estimate what a sensible budget looks like.
Key takeaways
- Small stores usually accept datacenter traffic; large retailers and marketplaces usually need residential.
- Structured data such as JSON-LD and internal APIs is lighter and more reliable than parsing HTML.
- Use sessions for category pagination and location-specific stock.
- Keep a per-store routing table and review it monthly.
- Monitor record counts per store to catch layout changes.
FAQ
Can I scrape Shopify stores with Webshare datacenter proxies?
Many Shopify stores accept datacenter traffic at reasonable rates, but some add bot protection. Test each store.
Do I need residential proxies for marketplaces?
Usually, yes. Large marketplaces are among the strictest targets.
How do I avoid scraping the same product twice?
Normalise URLs, strip tracking parameters, and key products by their ID or SKU.
Can I get stock levels by store?
Often, by setting the store or postcode in the session and keeping a sticky IP.
Should I render pages with a browser?
Only when data isn't in the HTML or a JSON endpoint. Rendering costs far more bandwidth.
How do I handle sites that change prices by location?
Use IPs in the right region and set location in the session explicitly.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.