Using Proxies in Scrapy: Middleware, Rotation and Best Practices

How Scrapy handles proxies

Scrapy's built-in HttpProxyMiddleware (enabled by default) routes requests through a proxy when you set request.meta['proxy'], or when http_proxy/https_proxy environment variables are present. Credentials in the proxy URL are converted into a Proxy-Authorization header automatically.

The simplest setup: per-request meta

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": "http://USER:PASS@gate.example.com:7777"})

    def parse(self, response):
        ...

With a rotating residential gateway, this alone gives you IP rotation, since the gateway assigns exits for you. See rotating residential proxies.

Environment variables

export https_proxy="http://USER:PASS@gate.example.com:7777"
export http_proxy="$https_proxy"
scrapy crawl products

Scrapy picks these up at startup. meta['proxy'] overrides them per request.

A custom rotating middleware

For static proxy lists, write a downloader middleware that assigns a proxy per request and retires ones that fail:

# middlewares.py
import random
from collections import defaultdict

class RotatingProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies
        self.failures = defaultdict(int)

    @classmethod
    def from_crawler(cls, crawler):
        path = crawler.settings.get("PROXY_LIST_FILE")
        with open(path) as f:
            lines = [l.strip() for l in f if l.strip()]
        proxies = []
        for l in lines:
            host, port, user, pw = l.split(":")
            proxies.append(f"http://{user}:{pw}@{host}:{port}")
        return cls(proxies)

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            healthy = [p for p in self.proxies if self.failures[p] < 5] or self.proxies
            request.meta["proxy"] = random.choice(healthy)

    def process_response(self, request, response, spider):
        if response.status in (403, 429) or b"captcha" in response.body.lower():
            self.failures[request.meta.get("proxy")] += 1
            retry = request.replace(dont_filter=True)
            retry.meta.pop("proxy", None)
            return retry
        return response

    def process_exception(self, request, exception, spider):
        self.failures[request.meta.get("proxy")] += 1

# settings.py
PROXY_LIST_FILE = "proxies.txt"
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 610,
}

In production, add a retry limit (for example via a counter in request.meta) so a page that's blocked everywhere doesn't loop forever.

Third-party packages

• scrapy-rotating-proxies provides rotation with ban detection via ROTATING_PROXY_LIST.
• Several proxy providers publish their own Scrapy middlewares.

Check that any package is maintained and compatible with your Scrapy version.

Settings that matter with proxies

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]
DOWNLOAD_TIMEOUT = 30
COOKIES_ENABLED = False  # unless you need sessions

Proxies spread load across IPs, but per-domain concurrency still matters, because sites watch total traffic. See rate limiting and proxies.

Sticky sessions in Scrapy

For paginated flows, keep the same session ID across a chain of requests by passing it in meta from one callback to the next:

def parse(self, response):
    sid = response.meta.get("sid")
    next_url = response.css("a.next::attr(href)").get()
    if next_url:
        yield response.follow(next_url, meta={"sid": sid,
            "proxy": f"http://customer-USER-session-{sid}:PASS@gate.example.com:7777"})

See sticky vs rotating sessions.

JavaScript-heavy sites

For pages that need rendering, integrate a browser via scrapy-playwright and configure the proxy in Playwright's launch or context options. See Playwright proxies.

Choosing a proxy strategy for Scrapy projects

Per-domain proxy routing

A small middleware can route specific domains through residential proxies and everything else through datacenter proxies:

from urllib.parse import urlparse

RESI_DOMAINS = {"bigretailer.example", "marketplace.example"}
RESI = "http://customer-USER-country-us:PASS@gate.example.com:7777"

class TieredProxyMiddleware:
    def __init__(self, dc_proxies):
        self.dc = dc_proxies
        self.i = 0

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("DC_PROXIES"))

    def process_request(self, request, spider):
        if "proxy" in request.meta:
            return
        host = urlparse(request.url).hostname or ""
        if any(host.endswith(d) for d in RESI_DOMAINS):
            request.meta["proxy"] = RESI
        else:
            request.meta["proxy"] = self.dc[self.i % len(self.dc)]
            self.i += 1

This keeps expensive residential bandwidth for the sites that need it. See proxy rotation strategies and cheap proxies.

Detecting soft blocks in Scrapy

Many blocks return HTTP 200 with a challenge page. Add validation in your parse callbacks or a middleware:

def looks_blocked(response):
    body = response.text.lower()
    return any(s in body for s in ("captcha", "access denied", "verify you are human"))

Retry with a new proxy when this returns true, and cap retries via a counter in request.meta.

Logging proxy performance

Record the proxy (or session) on each item or in stats, then review success by proxy and domain. Scrapy's stats collector can track custom counters, for example self.crawler.stats.inc_value(f"proxy/{tier}/success").

Common Scrapy proxy mistakes

• Forgetting dont_filter=True on retries, so duplicate filtering drops them.
• High CONCURRENT_REQUESTS_PER_DOMAIN that overwhelms targets regardless of proxy count.
• Cookies enabled across rotating IPs, linking requests that should look independent.
• No retry cap, causing infinite loops on permanently blocked pages.

See web scraping proxies and avoiding IP bans when scraping.

Scrapy with rotating residential: settings that work

# settings.py
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.75
RANDOMIZE_DOWNLOAD_DELAY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 20
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
RETRY_TIMES = 3
DOWNLOAD_TIMEOUT = 30
DEFAULT_REQUEST_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36"

These conservative settings suit protected sites; loosen them for lenient targets.

Monitoring a Scrapy crawl's proxy health

Watch these stats during a crawl:

• downloader/response_status_count/200 vs 403 and 429.
• retry/count and retry/max_reached.
• Your own counters for soft blocks.
• Items scraped per minute.

A rising share of 403/429 or soft blocks means slow down or move the domain to a higher proxy tier. See rate limiting and proxies.

Deployment notes

When deploying spiders to servers or a hosting platform, pass proxy credentials as environment variables or settings from the platform's secret store, not in code. If using IP whitelisting, make sure the deployment environment's outbound IP is whitelisted. See proxy authentication methods.

Per-proxy concurrency with download slots

Scrapy limits concurrency and delay per download slot, and by default a slot is the target domain. That's right for politeness towards a site, but with a static proxy list it means nothing stops several concurrent requests piling onto one IP.

Scrapy lets you choose the slot yourself through the download_slot meta key, and set limits per slot with the DOWNLOAD_SLOTS setting. Keying slots by proxy gives you per-IP limits:

# in your rotating middleware's process_request
proxy = self.pick_proxy()
request.meta["proxy"] = proxy
request.meta["download_slot"] = proxy  # one slot per proxy

# settings.py
CONCURRENT_REQUESTS_PER_DOMAIN = 2   # applies per slot, so per proxy here
DOWNLOAD_DELAY = 1

For named slots with their own values, DOWNLOAD_SLOTS takes a dictionary such as {"slot-name": {"concurrency": 1, "delay": 2}}.

Two cautions. First, once slots are keyed by proxy, per-domain politeness no longer comes for free; if many proxies hit one site, cap total concurrency with CONCURRENT_REQUESTS, or combine domain and proxy in the slot name. Second, the slot stays with a request through redirects, as Scrapy's settings documentation explains.

With a rotating gateway you usually don't need this; the gateway spreads load for you. See rate limiting and proxies and proxy rotation strategies.

Key takeaways

• Scrapy's HttpProxyMiddleware uses request.meta['proxy'] and converts URL credentials to headers automatically.
• A custom middleware can rotate static lists, track failures and route domains to different proxy tiers.
• Per-domain concurrency, delays and AutoThrottle matter regardless of proxy count.
• Detect soft blocks in responses and cap retries.
• Use scrapy-playwright for JavaScript-heavy pages. See python requests proxies.

FAQ

Does Scrapy support SOCKS5 proxies?

Not natively through HttpProxyMiddleware. Use an HTTP proxy, or a local forwarder that converts HTTP to SOCKS5.

Why are my Scrapy requests still getting 403s?

Likely headers, TLS fingerprint or per-domain rate, not just the IP. See avoiding IP bans when scraping.

Which proxies work best with Scrapy?

Datacenter for lenient sites, rotating residential for protected ones. See best proxies for web scraping.