Custom Credential Providers

Sometimes your credentials live somewhere the SDK does not know about.

A secrets broker like HashiCorp Vault, a bespoke identity service, or a Secrets Manager entry holding rotated keys - none of these are in the default chain. A custom credential provider teaches the SDK to fetch from that source and, crucially, to refresh before the credentials expire.

Reach for this only when a built-in provider genuinely does not fit. The built-ins are safer, and a hand-rolled provider is code you now own.

Recipe

Wrap a fetch-from-somewhere function so the SDK calls it for credentials and re-calls it to refresh. boto3 uses RefreshableCredentials; SDK v3 uses a provider function that returns an expiration.

# --- Python (boto3) ---
import boto3
from botocore.credentials import RefreshableCredentials
from botocore.session import get_session
 
def fetch():
    # Call Vault/your broker here; return the botocore metadata shape.
    creds = load_from_vault()  # your function
    return {
        "access_key": creds["accessKeyId"],
        "secret_key": creds["secretAccessKey"],
        "token": creds["sessionToken"],
        "expiry_time": creds["expiresAt"],  # ISO-8601 string
    }
 
refreshable = RefreshableCredentials.create_from_metadata(
    metadata=fetch(), refresh_using=fetch, method="custom-vault",
)
botocore_session = get_session()
botocore_session._credentials = refreshable
session = boto3.Session(botocore_session=botocore_session)
 
s3 = session.client("s3", region_name="us-east-1")

// --- TypeScript (AWS SDK v3) ---
import { S3Client } from "@aws-sdk/client-s3";
import type { AwsCredentialIdentityProvider } from "@aws-sdk/types";
 
// Returning `expiration` lets the SDK re-invoke this to refresh.
const fromVault: AwsCredentialIdentityProvider = async () => {
  const creds = await loadFromVault(); // your function
  return {
    accessKeyId: creds.accessKeyId,
    secretAccessKey: creds.secretAccessKey,
    sessionToken: creds.sessionToken,
    expiration: new Date(creds.expiresAt),
  };
};
 
const s3 = new S3Client({ region: "us-east-1", credentials: fromVault });

When to reach for this:

• Credentials come from a broker (Vault, CyberArk) the SDK cannot read natively.
• You mint short-lived keys from your own identity service.
• You need custom caching or refresh logic beyond what a profile expresses.
• You are bridging a legacy secret store into modern SDK code.

Working Example

A common concrete case: keys stored in AWS Secrets Manager that rotate on a schedule.

The provider reads the secret, and by returning an expiration it lets the SDK re-fetch after rotation without a restart.

# --- Python (boto3) ---
import json
from datetime import datetime, timezone, timedelta
import boto3
from botocore.credentials import RefreshableCredentials
from botocore.session import get_session
 
sm = boto3.client("secretsmanager", region_name="us-east-1")
 
def fetch():
    raw = sm.get_secret_value(SecretId="prod/app/aws-keys")["SecretString"]
    data = json.loads(raw)
    return {
        "access_key": data["accessKeyId"],
        "secret_key": data["secretAccessKey"],
        "token": data.get("sessionToken"),
        # Re-fetch in 15 minutes to pick up rotation.
        "expiry_time": (datetime.now(timezone.utc) + timedelta(minutes=15)).isoformat(),
    }
 
refreshable = RefreshableCredentials.create_from_metadata(
    metadata=fetch(), refresh_using=fetch, method="secrets-manager",
)
bs = get_session()
bs._credentials = refreshable
s3 = boto3.Session(botocore_session=bs).client("s3", region_name="us-east-1")

// --- TypeScript (AWS SDK v3) ---
import { S3Client } from "@aws-sdk/client-s3";
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
import type { AwsCredentialIdentityProvider } from "@aws-sdk/types";
 
const sm = new SecretsManagerClient({ region: "us-east-1" });
 
const fromSecret: AwsCredentialIdentityProvider = async () => {
  const out = await sm.send(new GetSecretValueCommand({ SecretId: "prod/app/aws-keys" }));
  const data = JSON.parse(out.SecretString ?? "{}");
  return {
    accessKeyId: data.accessKeyId,
    secretAccessKey: data.secretAccessKey,
    sessionToken: data.sessionToken,
    // Re-invoke in 15 minutes to pick up rotation.
    expiration: new Date(Date.now() + 15 * 60_000),
  };
};
 
const s3 = new S3Client({ region: "us-east-1", credentials: fromSecret });

What this demonstrates:

• RefreshableCredentials.create_from_metadata binds an initial fetch and a refresh_using callback.
• In SDK v3 the provider is a plain async function - no class or registration.
• Returning expiry_time / expiration is what enables automatic refresh.
• The bootstrap client (Secrets Manager here) needs its own credentials from the normal chain.

Deep Dive

The Two Refresh Models

boto3 and SDK v3 both refresh, but wire it differently.

• boto3: RefreshableCredentials holds an expiry_time and calls refresh_using when credentials are within an advisory window of expiry. It refreshes lazily, on the next request that needs credentials.
• SDK v3: a provider function is memoized; when the returned expiration nears, the SDK discards the cache and calls the function again. No expiration means "cache forever" - fine for static keys, wrong for rotating ones.

The Bootstrap Problem

Your provider often needs credentials to fetch credentials.

Reading Secrets Manager requires AWS auth, so the client that fetches the secret must itself authenticate through the normal chain - typically an instance or task role. Keep that bootstrap identity minimal: permission to read exactly one secret, nothing more.

Idempotency and Error Handling

The refresh callback can be invoked at any time, possibly concurrently.

Make it idempotent and fast. If the source is briefly unavailable, prefer raising a clear error over returning stale or partial credentials, and let the SDK's retry surface it. Cache within the provider only if your source is rate-limited.

When Not to Build One

Most "custom" needs are already covered.

Assuming a role? Use fromTemporaryCredentials / a profile. OIDC token? Use fromWebToken. An external CLI that prints credentials? boto3's credential_process and SDK v3's fromProcess already shell out for you. A custom provider is the last resort, not the first.

Gotchas

• No expiration returned - SDK v3 caches forever and never picks up rotation. Fix: always return an expiration for rotating credentials.
• Overwriting the wrong session credentials - setting _credentials on the default session can surprise other code. Fix: build a dedicated botocore_session for the custom provider.
• Bootstrap over-privilege - the fetching client has broad rights. Fix: scope it to reading the single secret it needs.
• Slow or blocking refresh - a heavy callback stalls requests. Fix: keep fetch fast and consider a short internal cache.
• Non-idempotent refresh - concurrent invocations mint or consume tokens twice. Fix: make the callback safe to call repeatedly.
• Reinventing a built-in - hand-rolling assume-role or web-identity logic. Fix: use fromTemporaryCredentials, fromWebToken, or credential_process first.

Alternatives

FAQs

Related

• The Credential Provider Chain, End to End - where a custom provider slots in.
• STS AssumeRole & Cross-Account Access - the built-in you should prefer for roles.
• Web Identity Federation & IRSA (EKS) - the built-in for OIDC sources.
• Production Configuration Best Practices - keeping custom auth code safe.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).