Using Proxies With Python Requests (and httpx, aiohttp)

The basics

Python's requests library takes proxies as a dictionary mapping URL schemes to proxy URLs:

import requests

proxy = "http://USER:PASS@gate.example.com:7777"
proxies = {"http": proxy, "https": proxy}

r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=30)
print(r.json())

Two points that trip people up:

• The "https" key is the scheme of the target URL, not the proxy. You almost always use an http:// proxy URL for both keys, because HTTPS targets are tunnelled through the HTTP proxy with CONNECT. See how proxies work.
• Always set a timeout. Without one, a stuck proxy can hang your script forever.

Sessions

Use a Session for connection reuse and default settings:

s = requests.Session()
s.proxies.update(proxies)
s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36"})
r = s.get("https://httpbin.org/ip", timeout=30)

Note: with rotating gateways, connection reuse can keep you on the same exit IP for multiple requests. If you need a new IP per request, use a new session ID in the username or a new Session.

Environment variables

requests respects HTTP_PROXY, HTTPS_PROXY and NO_PROXY:

export HTTPS_PROXY="http://USER:PASS@gate.example.com:7777"
export HTTP_PROXY="$HTTPS_PROXY"
export NO_PROXY="localhost,127.0.0.1"

Explicit proxies= arguments override environment variables. Set session.trust_env = False to ignore them.

Special characters in credentials

URL-encode usernames and passwords that contain special characters:

from urllib.parse import quote
proxy = f"http://{quote('user', safe='')}:{quote('p@ss:word', safe='')}@gate.example.com:7777"

SOCKS5

Install the extra: pip install "requests[socks]". Then:

proxy = "socks5h://USER:PASS@socks.example.com:1080"   # socks5h = DNS through the proxy
proxies = {"http": proxy, "https": proxy}

Use socks5h rather than socks5 to avoid DNS leaks. See SOCKS5 proxies.

Rotating proxies

With a rotating residential gateway, generate a new session ID for each request or batch:

import uuid

def rotating_proxies():
    u = f"http://customer-USER-country-us-session-{uuid.uuid4().hex[:8]}:PASS@gate.example.com:7777"
    return {"http": u, "https": u}

With a static list:

import random
PROXIES = [l.strip() for l in open("proxies.txt") if l.strip()]

def to_url(line):
    host, port, user, pw = line.split(":")
    return f"http://{user}:{pw}@{host}:{port}"

def random_proxies():
    u = to_url(random.choice(PROXIES))
    return {"http": u, "https": u}

See proxy rotation strategies.

Retries with backoff

import time

def get_with_retry(url, attempts=4):
    for n in range(attempts):
        try:
            r = requests.get(url, proxies=rotating_proxies(), timeout=30)
            if r.status_code == 200:
                return r
            if r.status_code == 429:
                time.sleep(2 ** n)
        except (requests.ProxyError, requests.ConnectTimeout, requests.ReadTimeout):
            pass
    raise RuntimeError("all attempts failed")

httpx

import httpx

with httpx.Client(proxy="http://USER:PASS@gate.example.com:7777", timeout=30) as client:
    print(client.get("https://httpbin.org/ip").json())

Recent httpx versions use proxy=; older versions used proxies=. httpx also supports HTTP/2 (http2=True) and async clients.

aiohttp (async)

import aiohttp, asyncio

async def main():
    async with aiohttp.ClientSession() as s:
        async with s.get("https://httpbin.org/ip", proxy="http://USER:PASS@gate.example.com:7777") as r:
            print(await r.json())

asyncio.run(main())

For SOCKS with aiohttp, use the aiohttp-socks package.

TLS fingerprints

Python HTTP libraries have TLS fingerprints that differ from browsers, which some bot protection checks. Libraries such as curl_cffi can impersonate browser TLS fingerprints and accept proxies:

from curl_cffi import requests as cffi_requests
r = cffi_requests.get("https://example.com", impersonate="chrome",
                      proxies={"https": "http://USER:PASS@gate.example.com:7777"})

See browser fingerprinting and proxies.

Common errors

See common proxy errors.

A reusable Python proxy client

import random, time, uuid
import requests

class ProxyClient:
    def __init__(self, gate, user, password, country="us", timeout=30):
        self.gate, self.user, self.password = gate, user, password
        self.country, self.timeout = country, timeout
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0 Safari/537.36",
            "Accept-Language": "en-US,en;q=0.9",
        }

    def _proxies(self, sticky=None):
        sid = sticky or uuid.uuid4().hex[:8]
        url = f"http://{self.user}-country-{self.country}-session-{sid}:{self.password}@{self.gate}"
        return {"http": url, "https": url}

    def get(self, url, attempts=4, sticky=None):
        for n in range(attempts):
            try:
                r = requests.get(url, proxies=self._proxies(sticky), headers=self.headers, timeout=self.timeout)
                if r.status_code == 200 and "captcha" not in r.text.lower():
                    return r
                if r.status_code == 429:
                    time.sleep(int(r.headers.get("Retry-After", 2 ** n)))
            except requests.RequestException:
                pass
            time.sleep(1.5 ** n + random.random())
        return None

client = ProxyClient("gate.example.com:7777", "customer-USER", "PASS")
resp = client.get("https://httpbin.org/ip")
print(resp.json() if resp else "failed")

Pass a fixed sticky value to keep the same IP across related requests (pagination, logged-in flows).

Async scraping at scale

For many concurrent requests, httpx.AsyncClient or aiohttp with a semaphore keeps concurrency under control:

import asyncio, httpx

async def fetch(client, url, sem):
    async with sem:
        r = await client.get(url, timeout=30)
        return url, r.status_code

async def main(urls):
    sem = asyncio.Semaphore(20)
    async with httpx.AsyncClient(proxy="http://USER:PASS@gate.example.com:7777") as client:
        return await asyncio.gather(*(fetch(client, u, sem) for u in urls))

With a rotating gateway, a single async client may reuse connections; create clients per batch or use session IDs if you need guaranteed rotation.

Choosing proxies for Python scraping

Datacenter for lenient sites; residential for protected ones; ISP for logged-in flows. See best proxies for web scraping and cheap proxies.

Choosing the right Python HTTP library for proxy work

For a quick script, requests is fine. For thousands of concurrent requests, httpx or aiohttp. For targets that fingerprint TLS, curl_cffi or a real browser via Playwright.

Debugging checklist

When a Python proxy request fails:

1. Print the exact proxy URL with the password masked, and check the scheme (http:// vs socks5h://).
2. Reproduce with curl using the same proxy. See curl proxy.
3. Check environment variables (env | grep -i proxy) that might override or conflict.
4. Set timeout=(10, 30) to separate connect and read timeouts, which tells you where it's hanging.
5. Catch and log the specific exception class (ProxyError, ConnectTimeout, ReadTimeout, SSLError).
6. Test against a simple endpoint (https://api.ipify.org) before your real target.

Keeping secrets out of code

import os
proxy = f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}@{os.environ['PROXY_HOST']}"

Load variables from a .env file with python-dotenv in development, and from your platform's secret store in production. Never commit proxy credentials to version control.

For proxy type selection, see best proxies for web scraping and residential proxies.

Proxy settings requests picks up without asking

By default, requests doesn't only read the proxies= you pass. With trust_env on, it also looks at the environment, and through Python's standard library that can include the operating system's proxy settings:

• Environment variables: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY, in upper or lower case.
• System settings on Windows and macOS: when no proxy variables are set, Python's urllib.request.getproxies() reads the Windows registry or the macOS system configuration, and requests uses the result.
• .netrc credentials for hosts listed there.

That explains some puzzling behaviour: a script that works on a server but goes through the office proxy on a laptop, or that ignores NO_PROXY set somewhere you forgot about. To see what Python thinks the proxies are:

import urllib.request, requests
print(urllib.request.getproxies())
print(requests.utils.get_environ_proxies("https://example.com"))

To make a session use only what you give it:

s = requests.Session()
s.trust_env = False
s.proxies = {"http": proxy, "https": proxy}

Explicit proxies= arguments still win over environment values, but trust_env = False removes the surprises entirely. The requests documentation covers proxies in detail. On machines where system proxies are set, see proxy setup on Windows and proxy setup on macOS.

FAQ

Why does requests ignore my proxy?

Usually a typo in the dict keys (they must be "http" and "https"), or NO_PROXY matching the host.

Should the proxy URL start with http or https?

Usually http://, even for HTTPS targets. Use https:// only if your provider offers TLS connections to the proxy itself.

What proxies are best for Python scraping?

Depends on the target. See best proxies for web scraping.