How to Use Geonode Proxies with Scrapy

How Scrapy uses proxies

Scrapy's built-in HttpProxyMiddleware reads a proxy URL from request.meta["proxy"] and, if the URL includes credentials, adds the Proxy-Authorization header for you. That means adding Geonode to a spider is a matter of setting the right proxy URL on each request, usually from a small downloader middleware so every request gets one.

Generate endpoints in the Geonode dashboard and keep credentials in environment variables (GEONODE_HOST, GEONODE_PORT, GEONODE_USER, GEONODE_PASS).

The quickest way: request meta

import os
import scrapy

def geonode_url(country=None):
    user = os.environ["GEONODE_USER"] + (f"-country-{country}" if country else "")
    return f"http://{user}:{os.environ['GEONODE_PASS']}@{os.environ['GEONODE_HOST']}:{os.environ['GEONODE_PORT']}"

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalog"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": geonode_url("us")})

The -country- suffix follows Geonode's documented username pattern; use your dashboard's format.

A downloader middleware for every request

# middlewares.py
import os

class GeonodeProxyMiddleware:
    def __init__(self, country):
        self.country = country

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("GEONODE_COUNTRY"))

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            user = os.environ["GEONODE_USER"] + (f"-country-{self.country}" if self.country else "")
            request.meta["proxy"] = (
                f"http://{user}:{os.environ['GEONODE_PASS']}"
                f"@{os.environ['GEONODE_HOST']}:{os.environ['GEONODE_PORT']}"
            )

# settings.py
GEONODE_COUNTRY = "us"
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.GeonodeProxyMiddleware": 350,
}

Priority 350 runs before the built-in proxy middleware (750), so the meta value is in place when it processes the request.

Routing domains to different products

Send protected domains through residential and lenient ones through cheaper datacenter traffic by choosing the proxy URL per domain in the middleware:

from urllib.parse import urlparse

RESIDENTIAL_DOMAINS = {"bigretailer.com", "marketplace.example"}

def process_request(self, request, spider):
    host = urlparse(request.url).hostname or ""
    if any(host.endswith(d) for d in RESIDENTIAL_DOMAINS):
        request.meta["proxy"] = self.residential_url
    else:
        request.meta["proxy"] = self.datacenter_url

This single change often cuts proxy costs dramatically. See Geonode datacenter proxies.

Sticky sessions for pagination

For paginated results or flows that depend on cookies, use a sticky endpoint for the whole sequence. Pass the proxy URL from the first request to follow-up requests through meta, so the chain stays on one exit.

Throttling, retries and bans

• AutoThrottle: AUTOTHROTTLE_ENABLED = True adapts request rate to the server's response times.
• Concurrency per domain: CONCURRENT_REQUESTS_PER_DOMAIN keeps you polite.
• Retries: RETRY_TIMES and RETRY_HTTP_CODES control built-in retries. A rotating endpoint gives retries a new IP automatically.
• Ban detection: add a middleware that inspects responses for CAPTCHA pages or missing expected elements and raises a retry, since blocks often return 200.

Bandwidth control

Scrapy fetches HTML by default and doesn't load images or scripts, which makes it naturally efficient on per-GB residential billing. Keep it that way: avoid downloading media, use HTTP caching during development (HTTPCACHE_ENABLED = True) so you don't re-fetch while writing parsers, and don't re-crawl unchanged pages. Geonode's residential bandwidth doesn't expire, so savings carry forward.

Troubleshooting

Choosing Geonode products for Scrapy projects

Scrapy is built for crawling many pages efficiently, which makes product choice and routing especially important.

Start each new domain on datacenter traffic and promote it only if success falls. Scrapy's stats collector makes it easy to count responses by status per domain; extend it with a custom stat for detected blocks.

A ban-detection middleware sketch

Blocks often return HTTP 200 with a challenge page, so status codes alone mislead Scrapy's retry logic. A small downloader middleware can inspect responses and turn block pages into retries:

• Check for a selector that should exist on real pages (a product title, a results list).
• Check for common challenge markers in the body.
• If a response looks blocked, return a new copy of the request with dont_filter=True and an incremented retry counter in meta, up to a limit.
• Increment a "blocked" stat per domain so you can see trends.

With a rotating Geonode endpoint, the retried request gets a new exit automatically.

Keeping crawls polite and cheap

• Respect robots.txt with ROBOTSTXT_OBEY = True unless you have a specific, legitimate reason not to.
• Use AutoThrottle and per-domain concurrency limits.
• Avoid re-crawling unchanged pages by storing fingerprints or last-modified values.
• Keep media out: don't enable image or file pipelines unless you need them.

Scrapy's HTML-only fetching already makes it one of the cheapest ways to use per-GB residential traffic. Combined with domain routing, many crawls run mostly on datacenter traffic. See Geonode for web scraping and ethical web scraping.

Quick checks before a long crawl

Run a short crawl of 50 to 100 pages per domain first, with LOG_LEVEL = "INFO" and the stats dump at the end. Check the response status counts, your custom "blocked" stat, average response time and bytes downloaded. If anything looks off, fix it before launching the full crawl. It's far cheaper to discover a misconfigured proxy or a blocking target in a 100-page test than in a 100,000-page run.

FAQ

Does Scrapy support authenticated proxies?

Yes. Put credentials in the proxy URL in request.meta["proxy"]; the built-in middleware handles the header.

How do I rotate IPs in Scrapy with Geonode?

Use a rotating endpoint. Each new connection gets a new exit.

Can I use different countries per spider?

Yes. Set a country per spider (or per request) in the middleware.

Do I need a browser with Scrapy?

Only for JavaScript-rendered pages. Consider the Geonode Scraper API or a headless browser integration for those.