Fail-closed proxy setup for crawlers and CI jobs on one local endpoint

A fail-closed proxy refuses a connection when it cannot route it through a working upstream, instead of letting it leave from the machine's own address. For a crawler or a CI job, that is the difference between a run that stops and says why, and a run that quietly completes from the office or build-box IP: the target sees an address you never meant to expose, and the results are no longer comparable with the rest. This post covers where proxy setups fail open, how one local endpoint that fails closed changes the picture, and the checks a job should still make for itself.

Where proxy setups fail open

Most fail-open incidents do not involve a proxy deciding to send traffic direct. They involve a request that never reached the proxy at all.

• A missing setting. A CI step that does not export HTTPS_PROXY, a cron job started without the shell profile, or a tool that reads a different variable from the one you set will connect directly without an error. The curl proxy cheat sheet lists which variables curl reads and how NO_PROXY exempts hosts.
• Fallback in code. A retry wrapper that drops the proxy after a few failures turns a proxy outage into direct traffic. It usually started as a convenience during development.
• DNS outside the tunnel. With socks5://, the client resolves the hostname itself and only the connection goes through the proxy; socks5h:// resolves through the proxy. SOCKS5 vs HTTP proxy explains the difference.
• Credentials scattered across tools. When every scraper, test and script holds its own proxy list, rotating a password or retiring an address means editing all of them, and the one that was missed keeps failing or falls back.

A single local endpoint addresses the last three: tools carry one setting that does not change, the proxy list lives in one place, and the endpoint decides what happens when no proxy works.

One local endpoint that refuses rather than falls back

ProxyDesk runs a SOCKS5 listener on 127.0.0.1:11080 and an HTTP listener on 127.0.0.1:18080, and spreads each new connection across the proxies you add or sign in for. A proxy that fails a connection is set aside, the connection is retried through up to three proxies in total, and the proxy that failed is re-tested in the background: after five minutes, then after waits that double up to 30 minutes. It returns once it passes.

When no proxy can take a connection, ProxyDesk refuses it. We stopped the only upstream in a local test and sent requests through both listeners:

Nothing reached the target from the machine's own address. Every one of those is an error a job can see: a non-zero exit code, an exception from the HTTP client, a failed status. Once the upstream was back and passed its re-test, the same commands succeeded without a restart.

Wiring a CI job to it

On a build box running ProxyDesk, export the variables from its Env snippet in the job, so every tool that honours them uses the endpoint:

export ALL_PROXY=socks5h://127.0.0.1:11080
export HTTP_PROXY=http://127.0.0.1:18080
export HTTPS_PROXY=http://127.0.0.1:18080

Then make the job prove where it exits before it does any work. This pre-flight asks the echo endpoint for the caller's address once directly and once through the local endpoint, and stops the job if the proxied request fails or comes back with the machine's own address:

#!/usr/bin/env bash
set -euo pipefail

ECHO=https://api.proxyforge.io/tools/echo
PROXY=${PROXY:-socks5h://127.0.0.1:11080}

ip_of() { sed -E 's/.*"ip":"([^"]+)".*/\1/'; }

direct=$(curl -sS --fail --max-time 15 "$ECHO" | ip_of)
proxied=$(curl -sS --fail --max-time 15 -x "$PROXY" "$ECHO" | ip_of) || {
  echo "no exit through $PROXY" >&2
  exit 1
}

if [ "$proxied" = "$direct" ]; then
  echo "exit IP $proxied is this machine's own address" >&2
  exit 1
fi
echo "exit IP $proxied"

We ran it against a local endpoint in three states: upstream down, endpoint not running, and an upstream that exits from the same address as the machine. Each one stopped with exit code 1 and said why. The check catches the failure no endpoint can: a job whose proxy setting never took effect.

If the build box is shared, set a local login in ProxyDesk's settings. SOCKS5 clients then have to authenticate, and HTTP clients without Proxy-Authorization receive 407, so only the jobs you configured can spend your proxies.

Crawlers: one setting, a pool managed elsewhere

A crawler pointed at the local address no longer needs a proxy list, a rotation middleware or its own health checks. curl, Python requests and Playwright each take a single proxy setting, and ProxyDesk generates the snippet for each with your ports filled in; Scrapy picks up the same HTTP_PROXY and HTTPS_PROXY variables the Env snippet exports. The ProxyDesk setup guide shows the snippets.

Two choices still belong to the crawl:

1. Rotation mode. The default gives each new connection the next usable proxy. The alternative stays on one proxy until it fails, for flows that must keep one exit IP; rotating vs sticky proxies covers which flows those are.
2. Protocol. The HTTP listener sends one forwarded request per upstream connection for plain http:// URLs, while HTTPS through CONNECT and everything over SOCKS5 behave normally. A crawl of many plain-HTTP pages is better pointed at the SOCKS5 listener.

Blocks from the target are a separate problem from dead proxies. A 403 or 429 comes back through a working proxy, so the endpoint passes it on rather than refusing; handling proxy 429 and 403 errors covers the retry policy.

Evidence after the run

When a run fails, the question is whether the proxies failed, the target refused, or the job never used the proxy. ProxyDesk keeps the last 500 connections of the session, with the destination, the upstream used, the result and the bytes, and Export CSV writes them with the columns time, destination, upstream, result, cause, error, bytes_up and bytes_down. A refused connection appears with its cause, so a run that failed closed can be told from one the target blocked. Export before the session rolls past 500 rows, and attach the file to the ticket. For metrics across many runs, proxy monitoring in production covers what to track.

What it does not cover

ProxyDesk carries TCP only, so UDP, DNS over UDP and HTTP/3 do not pass through it. It is a desktop app with no headless build, so it suits workstations and build boxes with a desktop session rather than server fleets; for clusters, a Kubernetes egress proxy applies the same one-setting idea at the pod level. And it fails closed only for traffic that reaches it, which is why the pre-flight belongs in the job.

The ProxyDesk page has screenshots of each screen and the platforms it runs on.