Rayobyte vs Webshare for Scrapy: Lists, Gateways and Routing

Scrapy on both

Scrapy reads request.meta["proxy"]. For Rayobyte, set a gateway URL; for Webshare, set the backbone URL or pick an IP from your list in a middleware. See Rayobyte with Scrapy, Webshare with Scrapy and Scrapy proxies.

A mixed middleware

import os, random
from urllib.parse import urlparse

WS_LIST = open(os.environ["WEBSHARE_LIST_FILE"]).read().split()
RY_RESI = os.environ["RAYOBYTE_RESI_PROXY"]
HARD = {"hard-shop.example", "big-marketplace.example"}

class MixedProxy:
    def process_request(self, request, spider):
        if "proxy" in request.meta:
            return
        if urlparse(request.url).hostname in HARD:
            request.meta["proxy"] = RY_RESI
        else:
            host, port, user, pw = random.choice(WS_LIST).split(":")
            request.meta["proxy"] = f"http://{user}:{pw}@{host}:{port}"

Cheap Webshare IPs handle easy domains; Rayobyte residential handles the hard ones.

Differences

Concurrency

Keep per-IP concurrency low on Webshare lists (many requests share each IP) and size CONCURRENT_REQUESTS to your Rayobyte plan.

With Geonode: Scrapy across all three.

A real-world example

A crawler indexes 2 million pages a month from 400 sites. Its middleware picks a random IP from a 1,500-IP Webshare list for most sites and routes 30 protected sites to Rayobyte residential. Scrapy stats count responses by provider and status, so the team can see in each crawl report where blocks happen and whether a site should move routes.

Useful Scrapy stats

• provider/webshare/200, provider/rayobyte/403 and so on, incremented in the middleware.
• Bytes downloaded per provider, to estimate cost.
• Retry counts per domain, to spot sites that need promotion.

Pros and cons for Scrapy

Rayobyte

• Every product, including the unblocker, fits Scrapy's proxy model.
• Residential cheaper at volume.

Webshare

• Cheap lists for broad crawls.
• Backbone gateway for rotation without a list.

Bottom line

A two-route middleware (Webshare by default, Rayobyte for protected domains) suits most Scrapy projects.

Rayobyte, Geonode and Webshare are three options among many. Our best proxies in 2026 guide explains how to match a proxy type to each task, and cheap proxies that work shows where budget providers fit.

Key takeaways

• Both plug into request.meta["proxy"].
• A middleware can mix cheap Webshare IPs with Rayobyte residential.
• Rayobyte's Web Unblocker fits Scrapy's proxy model.
• Keep per-IP concurrency low on lists.

FAQ

Do I need a rotation package with Webshare?

Not if you use the backbone or a simple middleware like the one above.

Can Scrapy use Rayobyte's Web Unblocker?

Yes, as a proxy URL.

Which is cheaper for big crawls?

Webshare lists on lenient sites; Rayobyte residential on protected sites at volume.

How do I add custom stats in Scrapy?

Call spider.crawler.stats.inc_value("provider/webshare/200") from your middleware.

Can I switch all traffic to one provider during an outage?

Yes; a setting such as FORCE_PROVIDER in your router makes it a one-line change.

Where should I store proxy credentials in Scrapy?

In environment variables read by settings.py, never in the repository.