Webshare vs Geonode for Scrapy: Middleware, Rotation and Cost at Scale

Scrapy sees proxies as one meta key

Scrapy reads each request's proxy from request.meta["proxy"], so supporting both Webshare and Geonode in one project is straightforward. The design question is which provider and product each request should use. Provider guides: Webshare with Scrapy and Geonode with Scrapy.

Gateway vs list

A list in a middleware gives per-IP control, which Scrapy's per-domain throttling pairs well with. A gateway is simpler and gives you a large pool without managing IPs.

Routing per domain to the right provider

# middlewares.py
import random

class ProviderRouter:
    def __init__(self, dc_list, residential_url, hard_domains):
        self.dc, self.res, self.hard = dc_list, residential_url, set(hard_domains)

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(s.getlist("WEBSHARE_DC_LIST"), s.get("GEONODE_RES_URL"), s.getlist("HARD_DOMAINS"))

    def process_request(self, request, spider):
        if request.meta.get("proxy"):
            return
        host = request.url.split("/")[2]
        request.meta["proxy"] = self.res if host in self.hard else random.choice(self.dc)

This sends easy domains through a cheap Webshare datacenter list and hard domains through Geonode residential. Swap the assignments if your testing shows different results.

Retries with a fresh IP

On a gateway with rotation, every retry gets a new IP automatically. With a list, clear meta["proxy"] on retry so your middleware picks another IP. On both, add 403 and 429 to RETRY_HTTP_CODES carefully and back off, so you don't burn bandwidth on a target that's blocking you.

Sticky crawls

For paginated or logged-in paths, pin one session per path:

• Webshare: a session number in the backbone username.
• Geonode: a session ID with a lifetime long enough for the crawl path (up to 24 hours).

Pair each with Scrapy's cookiejar meta key so cookies and IP stay together.

Concurrency settings

CONCURRENT_REQUESTS = 64
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
DOWNLOAD_TIMEOUT = 30

Keep total concurrency through Webshare below your plan's thread limit, and ramp up gradually on residential gateways while watching error rates.

Cost at crawl scale

Verdict

Scrapy works equally well with both. The win comes from routing: cheap Webshare lists for the bulk, Geonode residential for the hard minority.

Settings for a two-provider project

# settings.py
WEBSHARE_DC_LIST = ["http://USER:PASS@IP1:PORT", "http://USER:PASS@IP2:PORT"]  # load from the API in production
GEONODE_RES_URL = "http://GN_USER:GN_PASS@GN_HOST:GN_PORT"
HARD_DOMAINS = ["bigshop.example", "travel.example"]
DOWNLOADER_MIDDLEWARES = {"myproject.middlewares.ProviderRouter": 350}

Load credentials from environment variables rather than hard-coding them.

Reporting per provider

Use Scrapy stats to count responses by provider and status, for example provider/webshare/200 and provider/geonode/403. At the end of each crawl, these numbers tell you whether routing is working and where blocks happen.

Handling a provider outage

If one provider returns errors across all domains, temporarily route everything through the other. A simple setting such as FORCE_PROVIDER = "geonode" in your router makes this a quick change during an incident.

Cost tracking

Multiply Geonode residential bytes (from response sizes) by its per-GB rate, and divide Webshare's monthly plan cost by the number of requests it served. Comparing cost per item scraped per provider shows whether your routing is optimal.

A real-world example

A company runs 40 Scrapy spiders for competitor catalogs. A shared ProviderRouter middleware reads a YAML file of domain routes. Thirty-four spiders' domains use Webshare datacenter lists; six use Geonode residential with cookiejar sessions for pagination. Scrapy stats are exported per provider after each run, and a weekly report flags domains whose block rate rose, prompting a re-test on the other provider.

Before you commit to any provider, it helps to see the wider market: our guide to the best proxies in 2026 compares proxy types by job, and cheap proxies explains how to keep costs down without buying IPs that get blocked.

Key takeaways

• Scrapy uses meta["proxy"], so a routing middleware can send each domain to either provider.
• Datacenter lists suit bulk crawling; residential gateways suit protected domains.
• Clear the proxy on retries with lists so each retry gets a new IP.
• Track Scrapy stats per provider to tune routing.
• Keep a force-provider setting for incidents.

FAQ

Can one Scrapy project use two proxy providers?

Yes. Set meta["proxy"] per request based on domain or job.

Does Scrapy support SOCKS5?

Not natively. Use HTTP proxy URLs with both providers.

Which is cheaper for large Scrapy crawls?

For lenient targets, Webshare. For protected targets, usually Geonode.

Does Scrapy's HttpProxyMiddleware work with both providers?

Yes. It reads credentials from the proxy URL in meta["proxy"].

How do I test a provider for one spider only?

Override the route for that spider's domains in settings and compare stats.

Can Scrapy rotate Geonode sessions?

Yes. Generate a session ID per crawl path and include it in the proxy username.