How to set up a Python Requests proxy (auth, SOCKS5, retries)

To use a Python Requests proxy, pass a proxies dictionary with both an http and an https key pointing at the same proxy URL, or set HTTP_PROXY and HTTPS_PROXY in the environment. The key names refer to the scheme of the target URL, not the proxy, so forgetting the https key silently sends every HTTPS request direct. Production code also needs explicit timeouts, a retry policy, and a check that traffic really leaves through the proxy.

This guide covers each of those pieces with code that runs against the current releases of Requests and urllib3. Every example reads the proxy URL from an environment variable, PROXY_URL, shaped like http://USERNAME:[email protected]:PORT. Your dashboard generates the exact string for the country and session mode you pick; do not hand-assemble it from an article.

The minimal Python Requests proxy setup

import os

import requests

proxy_url = os.environ["PROXY_URL"]
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxies,
    timeout=(5, 30),
)
response.raise_for_status()
print(response.json())

Three details in this snippet matter more than they look:

• Both keys are set. proxies["https"] governs requests to https:// URLs. Requests does not warn when a scheme has no entry; it simply connects directly.
• The proxy URL uses http://. For an HTTPS target, Requests sends a CONNECT to the proxy and runs TLS to the destination inside that tunnel. The proxy sees the hostname and port, not the page content. Use an https:// proxy URL only if your provider documents a TLS listener on the gateway.
• The timeout is explicit. Requests has no default timeout. Without one, a stalled upstream can hold a worker forever. The tuple is (connect, read): keep the connect timeout short, because a proxy that cannot accept a connection in five seconds is not going to recover in thirty.

If you are setting up credentials for the first time, set up a proxy and verify your first request walks through the account side and the curl checks that should pass before any Python is involved.

Credentials with special characters

The username and password sit inside a URL, so any character with meaning in a URL (@, :, /, #, %) must be percent-encoded. An unencoded @ in a password is the most common cause of a proxy that "rejects valid credentials". If your credentials come from a secrets store as separate values, build the URL with urllib.parse.quote and safe='':

import os
from urllib.parse import quote

username = quote(os.environ["PROXY_USERNAME"], safe="")
password = quote(os.environ["PROXY_PASSWORD"], safe="")
proxy_url = f"http://{username}:{password}@gateway.proxyforge.io:PORT"

Replace PORT with the port from your dashboard, or store the host and port in the environment as well. Requests decodes the percent-encoding before it builds the Proxy-Authorization header, so the gateway receives the original characters.

If you would rather not ship credentials to every worker at all, allowlisting the egress IP of your fleet is the alternative. IP allowlist vs username and password covers when each mode is the better operational choice.

Environment variables, NO_PROXY and trust_env

Requests reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY (and their lowercase forms) whenever trust_env is true, which is the default. That lets you configure a Python Requests proxy for scripts and containers without touching code:

export PROXY_URL="http://USERNAME:[email protected]:PORT"
export HTTP_PROXY="$PROXY_URL"
export HTTPS_PROXY="$PROXY_URL"
export NO_PROXY="localhost,127.0.0.1,.internal.example.com"

NO_PROXY keeps internal traffic, such as metadata endpoints, health checks and your own APIs, off the proxy and off your bandwidth bill.

The trap is precedence. With trust_env enabled, environment proxies are merged into each request's settings before the Session's own proxies are applied, so an HTTPS_PROXY left in a container image overrides session.proxies. We confirmed this against Requests 2.34: a stray HTTPS_PROXY won over a Session configured in code. The order is:

Pick one source of truth per service. If proxy settings live in code, set session.trust_env = False so the environment cannot change them. Note that this also stops Requests reading .netrc and the REQUESTS_CA_BUNDLE variable.

Sessions, connection reuse and rotation

A requests.Session pools connections through urllib3. That is what you want for throughput, and it has a consequence for proxies that is easy to miss: for HTTPS targets, a pooled connection is a pooled CONNECT tunnel. On most gateways the exit address is chosen when the tunnel opens, so consecutive requests that reuse a pooled connection usually leave from the same IP, even on an endpoint configured to rotate per request.

That gives you two practical patterns:

• You want one identity for a sequence of requests (a login, a paginated listing, a checkout flow). Use a sticky session from your provider and one Session object per identity. The pool keeps the tunnel open, and the sticky session keeps the exit stable if the tunnel is re-established.
• You want a different IP per request. Use the rotating endpoint and do not rely on a long-lived pool; a fresh connection per request is what lets the gateway rotate.

The difference between the two modes, and how long a sticky session should last for a given target, is covered in rotating vs sticky proxies.

For concurrency, Requests does not document Session as thread-safe. Give each worker thread its own Session, and size the pool with HTTPAdapter(pool_maxsize=...) so a thread doing parallel work is not throttled by the default of ten connections per host. Beyond a few dozen concurrent requests per process, an async client is usually the better tool; see the httpx section below.

Timeouts and retries with urllib3

HTTPAdapter accepts a urllib3 Retry object. This is the one place in Requests where retries, backoff and Retry-After handling are implemented properly, so use it instead of a hand-written loop:

import os

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


def build_session(proxy_url: str) -> requests.Session:
    retry = Retry(
        total=4,
        connect=3,
        read=2,
        status=3,
        backoff_factor=0.5,
        backoff_jitter=0.25,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}),
        respect_retry_after_header=True,
    )
    adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=20)

    session = requests.Session()
    session.trust_env = False
    session.proxies = {"http": proxy_url, "https": proxy_url}
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session


session = build_session(os.environ["PROXY_URL"])
response = session.get("https://api.ipify.org?format=json", timeout=(5, 30))
response.raise_for_status()
print(response.json())

Some choices worth making deliberately:

1. Retry 429 and 5xx, not 403 or 407. A 407 means the gateway rejected your credentials, and retrying only burns time. A 403 from the target is usually a block that a retry through the same exit will not fix. Proxy 403 and 429 errors explains how to tell a target-side block from a proxy-side failure.
2. Respect Retry-After. urllib3 honors it for 429 and 503 by default. A site that tells you when to come back is giving you the cheapest possible fix.
3. Keep writes out of the retry list. allowed_methods above excludes POST and PUT so a timed-out form submission is not sent twice.
4. Retries reuse the pool. With a rotating endpoint and a pooled connection, a retry may go out through the same exit that was just rate-limited. If you retry on 429, prefer a fresh connection or a new sticky session for the retry.
5. backoff_jitter needs urllib3 2. Drop that argument if you are pinned to urllib3 1.26.

SOCKS5 with remote DNS

Every ProxyForge line also accepts SOCKS5. Requests needs the optional dependency:

pip install "requests[socks]"

Then use a SOCKS URL in the same proxies dictionary:

import os

import requests

socks_url = os.environ["SOCKS_PROXY_URL"]
proxies = {"http": socks_url, "https": socks_url}

response = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=(5, 30))
print(response.json())

The scheme decides where DNS happens. With socks5://, your machine resolves the hostname and sends the proxy an IP address, which leaks the lookup to your local resolver and can pick a CDN node near you rather than near the exit. With socks5h://, the hostname goes to the proxy and is resolved at the exit. For geo-targeted work, use socks5h://.

For most HTTP scraping there is no advantage to SOCKS5 over an HTTP proxy; the HTTPS tunnel is equally opaque either way. Reach for SOCKS when the rest of your stack already speaks it, or when you need to carry traffic that is not HTTP.

Verifying the exit IP

A broken Python Requests proxy configuration rarely raises an error. The request succeeds, just from the wrong address. Make the check part of your job start-up rather than something you run once by hand:

import os

import requests

IP_ECHO_URL = "https://api.ipify.org?format=json"


def exit_ip(session: requests.Session) -> str:
    response = session.get(IP_ECHO_URL, timeout=(5, 15))
    response.raise_for_status()
    return response.json()["ip"]


direct = requests.Session()
direct.trust_env = False

proxied = requests.Session()
proxied.trust_env = False
proxied.proxies = {"http": os.environ["PROXY_URL"], "https": os.environ["PROXY_URL"]}

local_ip = exit_ip(direct)
proxy_ip = exit_ip(proxied)
if proxy_ip == local_ip:
    raise SystemExit(f"Traffic is not going through the proxy (exit IP {proxy_ip})")
print(f"Exit IP through proxy: {proxy_ip}")

Log the exit IP alongside each job's results. When a target starts returning odd content, being able to answer "which address fetched this" turns a guess into a lookup. If you target a country, also check the exit against a geolocation source you trust, at country level; city-level results vary between databases.

Async and high concurrency: httpx

If you need hundreds of concurrent requests, an async client is simpler than a thread pool. In current httpx (0.28), the argument is proxy=, singular; the older proxies= argument was removed.

import asyncio
import os

import httpx


async def main() -> None:
    timeout = httpx.Timeout(30.0, connect=5.0)
    limits = httpx.Limits(max_connections=20)
    async with httpx.AsyncClient(proxy=os.environ["PROXY_URL"], timeout=timeout, limits=limits) as client:
        responses = await asyncio.gather(
            *(client.get("https://api.ipify.org?format=json") for _ in range(5))
        )
    for response in responses:
        print(response.json())


asyncio.run(main())

Unlike Requests, httpx applies a default timeout of five seconds, which is short for proxied traffic to slow targets; set it explicitly. SOCKS needs pip install "httpx[socks]". The same connection-reuse caveat applies: max_connections bounds both your concurrency and the number of tunnels, and therefore the number of distinct exits you see from a rotating endpoint at once. The httpx proxy documentation covers per-scheme routing with mounts if you need it.

Which proxy line fits a Requests workload

Requests fetches exactly the URLs you ask for, with no images, fonts or scripts, so bandwidth per page is low and per-GB lines are economical. The choice depends on the target:

If you are unsure, residential vs ISP vs datacenter proxies compares them in more depth.

Next steps with ProxyForge

Every ProxyForge line accepts HTTP, HTTPS and SOCKS5 with either username and password or an IP allowlist, so the code above works unchanged across lines; only the connection string differs. The dashboard generates that string for the country and session mode you choose.

Billing is pay-as-you-go from a prepaid wallet starting at 1 GB or 1 IP; current rates are on the pricing page. If you are moving an existing Requests pipeline from another provider, the migration process mirrors your current endpoint and session syntax so the code does not change during the parallel run.