Using Rayobyte Proxies With Scrapy

How Scrapy uses proxies

Scrapy's built-in HttpProxyMiddleware reads the proxy from request.meta["proxy"]. Put the full Rayobyte proxy URL there, credentials included, and Scrapy handles the rest:

yield scrapy.Request(url, meta={"proxy": "http://USERNAME:PASSWORD@HOST:PORT"})

Copy the values from the dashboard. Our general Scrapy proxies guide covers Scrapy proxy concepts.

A routing middleware

Most projects shouldn't hard-code one proxy. A tiny downloader middleware can route each domain to the cheapest Rayobyte product that works:

# middlewares.py
import os
from urllib.parse import urlparse

TIERS = {
    "dc": os.environ["RAYOBYTE_DC_PROXY"],
    "isp": os.environ["RAYOBYTE_ISP_PROXY"],
    "resi": os.environ["RAYOBYTE_RESI_PROXY"],
}
ROUTES = {"hard-shop.example": "resi", "medium-shop.example": "isp"}

class RayobyteProxyMiddleware:
    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            host = urlparse(request.url).hostname or ""
            request.meta["proxy"] = TIERS[ROUTES.get(host, "dc")]

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RayobyteProxyMiddleware": 350,
}

Priority 350 runs before Scrapy's own proxy middleware (750). The tiers map to rotating datacenter, rotating ISP and Premium Residential. See Rayobyte for web scraping for the routing logic.

Sticky sessions for pagination

When a listing needs the same IP across pages, generate a session ID once and pass it along in meta:

import uuid

def start_requests(self):
    sid = uuid.uuid4().hex[:10]
    proxy = self.settings["RAYOBYTE_STICKY_TEMPLATE"].format(session=sid)
    yield scrapy.Request(self.start_url, meta={"proxy": proxy, "sid": sid})

def parse(self, response):
    # ... extract items ...
    next_page = response.css("a.next::attr(href)").get()
    if next_page:
        yield response.follow(next_page, meta={"proxy": response.meta["proxy"], "sid": response.meta["sid"]})

The template comes from the dashboard, with {session} where the session ID goes. Premium Residential sessions last up to 120 minutes and can end early; see Rayobyte sticky sessions.

Retries

Scrapy's RetryMiddleware retries on connection errors and certain status codes. Add your block codes and keep the count modest:

RETRY_ENABLED = True
RETRY_TIMES = 2
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]

For blocks that return 200 with a CAPTCHA page, check the response content in your spider and re-yield the request with dont_filter=True and a new session.

Concurrency

CONCURRENT_REQUESTS = 64
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_TIMEOUT = 30
AUTOTHROTTLE_ENABLED = True

Keep CONCURRENT_REQUESTS under your plan's limit. Premium Residential lists unlimited concurrency; Scale Residential and monthly rotating ISP plans allow one request per dollar of plan. See concurrency and threads.

Saving bandwidth

• Enable compression (on by default) and avoid downloading binary files you don't need.
• Use sitemaps (SitemapSpider) for discovery.
• Turn on HTTPCACHE_ENABLED during development so you don't refetch the same pages while writing parsers.

JavaScript pages

Scrapy doesn't run JavaScript. For pages that need it, use scrapy-playwright with the Rayobyte proxy in the Playwright launch options, or send those URLs to the Scraping API. See Rayobyte with Playwright.

How it compares

The middleware works the same for Geonode and Webshare with their URLs. See Rayobyte vs Geonode for Scrapy, Rayobyte vs Webshare for Scrapy and all three for Scrapy.

Before you commit to any provider, it helps to see the wider market: our guide to the best proxies in 2026 compares proxy types by job, and cheap proxies explains how to keep costs down without buying IPs that get blocked.

Key takeaways

• Set request.meta["proxy"] to a Rayobyte proxy URL.
• A small middleware routes each domain to the right Rayobyte product.
• Pass a session ID through meta for paginated crawls.
• Keep concurrency under plan limits and retry with fresh sessions on blocks.

FAQ

Do I need scrapy-rotating-proxies with Rayobyte?

No. Rayobyte's rotating gateways rotate for you. Use a list-rotation package only with static IP lists.

How do I use Rayobyte static IPs in Scrapy?

Load the list from the dashboard and pick one per request in your middleware, keeping per-IP rates low.

Can Scrapy use SOCKS5 proxies?

Not natively. Use HTTP proxies with Scrapy.