Managing proxy credentials in Vault and AWS Secrets Manager

Proxy credentials belong in a secrets manager, read by the application at startup, cached in memory and refreshed on a schedule or when the gateway rejects them. They should never appear in source code, container images, CI output or logs, and each team or pipeline should read its own secret through a policy that grants nothing else. This guide shows how to do that with HashiCorp Vault and AWS Secrets Manager, with Python loaders you can drop into a service.

A proxy credential is easy to underestimate. It is one string, http://USERNAME:[email protected]:PORT, and it looks less sensitive than a database password. But anyone holding it can spend your prepaid balance and send traffic that the provider will attribute to your account, so it deserves the same handling as any other production secret.

What to store and how to name it

Store the full connection string the provider generates, as one field in a JSON secret, rather than separate username, password, host and port fields that the application reassembles. The dashboard's generated string already has any special characters percent-encoded and the targeting options applied; reassembly is where those details get lost.

{
  "proxy_url": "http://USERNAME:[email protected]:PORT",
  "owner": "pricing-team",
  "issued": "2026-09-30"
}

The non-secret fields cost nothing and help during an incident: they tell whoever is rotating the value whom to call and how old it is.

Use one secret per consumer boundary, and make the path say which boundary it is:

A consistent hierarchy is what makes least-privilege policies short. A policy that grants scraping/pricing/* is readable in a review; a list of twenty unrelated secret names is not.

Keeping proxy credentials out of code and logs

Most leaks of proxy credentials are not breaches. They are the value turning up somewhere it was never meant to be stored. Check each of these:

• Source and images. No credential in the repository, in a Dockerfile ENV or ARG, or in a baked configuration file. Build arguments are recorded in image history.
• CI output. Pipelines that echo environment variables, print rendered manifests or run with shell tracing (set -x) will print the URL.
• Application logs. Any log line that includes the proxy URL, a request's configuration or an exception that formats either. The URL carries the password in its userinfo section.
• HTTP debug output. Python's http.client debug mode prints the raw CONNECT request, including the Proxy-Authorization header, and writes it with print, which bypasses logging filters entirely. Never enable it in production.
• Error trackers. Tools that capture local variables in stack frames will capture the proxy URL if it is in scope. Configure their scrubbing rules for the variable names you use.

Proxy-Authorization with Basic authentication is base64, not encryption. It is defined in RFC 9110 and anyone who can read the header can decode it. With an http:// proxy URL, that header travels unencrypted to the gateway; the TLS session to the target runs inside the tunnel, but the tunnel request does not. If that matters in your environment, use IP allowlist authentication from a controlled egress address, or a TLS connection to the proxy where the provider documents one. IP allowlist vs username and password compares the two modes.

As a last line of defense, redact at the log formatter rather than at each call site. A formatter sees the final text, including formatted exceptions:

import logging
import re

USERINFO = re.compile(r"(?<=://)[^/@\s:]+:[^/@\s]+@")
PROXY_AUTH = re.compile(r"(Proxy-Authorization['\"]?\s*[:=]\s*['\"]?)[^'\"\r\n,}]+", re.IGNORECASE)


def redact(text: str) -> str:
    text = USERINFO.sub("***:***@", text)
    return PROXY_AUTH.sub(r"\1[redacted]", text)


class RedactingFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        return redact(super().format(record))


handler = logging.StreamHandler()
handler.setFormatter(RedactingFormatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
logging.basicConfig(level=logging.INFO, handlers=[handler])

This replaces the userinfo in any URL and the value of a Proxy-Authorization header, whether it appears as a raw header line or inside a printed dictionary. It does not replace the discipline of not logging the value in the first place.

Loading proxy credentials from AWS Secrets Manager in Python

Read the secret once, keep it in memory, and refresh it on a timer and whenever the gateway rejects it. The small cache below is shared by both stores in this guide; the loader is the only part that differs:

import json
import os
import threading
import time
from typing import Callable

import boto3
import requests


class CachedSecret:
    def __init__(self, loader: Callable[[], str], ttl_seconds: float = 300.0) -> None:
        self._loader = loader
        self._ttl = ttl_seconds
        self._lock = threading.Lock()
        self._value: str | None = None
        self._loaded_at = 0.0

    def get(self, force_refresh: bool = False) -> str:
        with self._lock:
            expired = time.monotonic() - self._loaded_at > self._ttl
            if self._value is None or expired or force_refresh:
                self._value = self._loader()
                self._loaded_at = time.monotonic()
            return self._value


secrets_client = boto3.client("secretsmanager")


def load_from_secrets_manager() -> str:
    response = secrets_client.get_secret_value(SecretId=os.environ["PROXY_SECRET_ID"])
    return json.loads(response["SecretString"])["proxy_url"]


proxy_secret = CachedSecret(load_from_secrets_manager)


def _get(url: str, proxy_url: str) -> requests.Response:
    proxies = {"http": proxy_url, "https": proxy_url}
    response = requests.get(url, proxies=proxies, timeout=(5, 30))
    if response.status_code == 407:
        raise requests.exceptions.ProxyError("407 Proxy Authentication Required")
    return response


def fetch(url: str) -> requests.Response:
    try:
        return _get(url, proxy_secret.get())
    except requests.exceptions.ProxyError:
        return _get(url, proxy_secret.get(force_refresh=True))

The environment carries only the secret's name (PROXY_SECRET_ID), which is safe to print. Credentials for the AWS call itself come from the execution role, never from a key pair in the environment.

The 407 handling covers both ways a rejected credential surfaces in Requests: for HTTPS targets, the gateway refuses the CONNECT and Requests raises ProxyError; for plain HTTP targets, the 407 comes back as a response. Either way the loader runs once more and the request is retried with the fresh value. If the retry also fails, the error propagates, which is correct: a credential that is wrong in the store is an incident, not something to retry in a loop.

If you prefer a maintained cache, AWS publishes a Python caching client for Secrets Manager that does the same time-based refresh. Keep the forced refresh on 407 either way.

Loading proxy credentials from Vault KV v2 with hvac

The Vault loader authenticates, reads the latest version from the KV v2 engine, and returns the same field. In Kubernetes, the Kubernetes auth method lets a pod log in with its service account token, so there is no Vault token to distribute. This reuses CachedSecret from the previous section:

import os

import hvac

SERVICE_ACCOUNT_TOKEN = "/var/run/secrets/kubernetes.io/serviceaccount/token"


def load_from_vault() -> str:
    client = hvac.Client(url=os.environ["VAULT_ADDR"])
    with open(SERVICE_ACCOUNT_TOKEN) as token_file:
        client.auth.kubernetes.login(role=os.environ["VAULT_ROLE"], jwt=token_file.read())
    secret = client.secrets.kv.v2.read_secret_version(
        path="scraping/pricing/proxy",
        mount_point="secret",
        raise_on_deleted_version=True,
    )
    return secret["data"]["data"]["proxy_url"]


proxy_secret = CachedSecret(load_from_vault)

KV v2 responses nest the payload under data.data, with version metadata under data.metadata. Passing raise_on_deleted_version=True makes a deleted latest version an error rather than an empty result, and silences hvac's warning about the default changing in a future release. The KV v2 documentation covers versioning and soft deletion.

One trap applies to both loaders: hvac and boto3 both honor HTTPS_PROXY. If you set the proxy globally in the environment, calls to Vault and to AWS will try to go through the scraping proxy too. Pass the proxy explicitly in code, as above, or add the Vault address and .amazonaws.com to NO_PROXY.

Caching, refresh and startup behavior

A few choices determine whether the secrets store becomes a dependency you notice:

1. Load at startup and fail fast. A worker that cannot read its credential should crash on boot, where the orchestrator reports it, not an hour later on the first request.
2. Refresh on a timer measured in minutes. Five to fifteen minutes is typical. Per-request reads add latency, cost (Secrets Manager bills per API call) and a hard dependency on the store for every request.
3. Refresh on rejection. A 407 is the gateway telling you the cached value is no longer valid. Refresh once, then fail.
4. Share the cache per process. Every thread in the process reads the same object; the lock in CachedSecret prevents a stampede of refreshes when the timer expires under load.
5. Consider an agent for fleets. Vault Agent or a sidecar that renders the secret to a file, or a Kubernetes controller that syncs it into a native Secret, moves the store interaction out of application code. Kubernetes egress proxy setup covers the Kubernetes side, including restarting pods after a rotation.

Least-privilege access policies

Each workload should be able to read its own proxy secret and nothing else. For AWS Secrets Manager, attach a policy like this to the workload's role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadPricingProxySecret",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:eu-central-1:111122223333:secret:scraping/pricing/proxy-??????"
    },
    {
      "Sid": "DecryptThroughSecretsManagerOnly",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-central-1:111122223333:key/KEY_ID",
      "Condition": {
        "StringEquals": {
          "kms:ViaService": "secretsmanager.eu-central-1.amazonaws.com"
        }
      }
    }
  ]
}

The -?????? suffix matches the six random characters Secrets Manager appends to every secret ARN, without matching a longer name that merely starts with the same prefix. The KMS statement is only needed if the secret is encrypted with a customer managed key; the condition restricts decryption to requests made through Secrets Manager. The principal that rotates the secret needs secretsmanager:PutSecretValue instead, and no workload role should have it.

The Vault equivalent is a policy with one path and one capability. Vault accepts policies in JSON as well as HCL:

{
  "path": {
    "secret/data/scraping/pricing/proxy": {
      "capabilities": ["read"]
    }
  }
}

Note the data/ segment: KV v2 policies address the API path, not the path you pass to the CLI. Bind the policy to a Kubernetes auth role scoped to the team's namespace and service account.

Per-team credentials and a rotation workflow

Separate secrets per team give you separate audit trails (CloudTrail or a Vault audit device records every read), separate blast radius and independent rotation. How far the provider lets you separate the credentials themselves depends on its account model, so ask. On ProxyForge, credentials and the gateway address are issued per account in the dashboard, and people are managed through an organization with three roles: Owner, User and Read only. Only Owners manage API keys, webhooks and members, which makes the Owner the natural holder of the rotation procedure. Proxy team access covers how to map those roles onto a platform team.

A rotation that does not cause an outage follows the same steps in either store:

1. Obtain the new credential and test it directly with curl -x from a machine that uses the same egress path as production.
2. Write it as a new version. In Secrets Manager, aws secretsmanager put-secret-value --secret-id scraping/pricing/proxy --secret-string file://new-proxy-secret.json moves the AWSCURRENT label to the new version and keeps the old one as AWSPREVIOUS. In Vault, vault kv put -mount=secret scraping/pricing/proxy @new-proxy-secret.json creates a new version. Delete the local file afterwards.
3. Let consumers pick it up. Timer-based caches converge within one refresh interval; services that received the value as an environment variable need a restart.
4. Watch 407 rates until every consumer has moved.
5. Revoke the old credential with the provider only then, and record the rotation.

If the provider can keep the old and new credentials valid at the same time, steps 3 to 5 need no coordination. If it cannot, schedule the rotation for a quiet window and keep the refresh-on-407 path in place, which limits the impact to one failed request per worker. This is a good question for your proxy vendor security questionnaire, alongside how the vendor itself stores and logs credentials.

Using this with ProxyForge

ProxyForge credentials drop straight into this pattern: copy the connection string the dashboard generates for your chosen country and session mode into the secret, and the loaders above work unchanged across residential, mobile, ISP and datacenter lines. Every line also supports IP allowlist authentication, which removes the password from the request path entirely when your workloads leave from fixed addresses.

For the security review itself, our sourcing page describes how the addresses behind those credentials are acquired and audited, and our data processing agreement is available before you sign. Talk to us through the contact page if your review needs more than the published material.