Debugging & Incident Response Best Practices

These are the defaults every AWS-backed service should set explicitly, drawn from the recurring defect patterns covered earlier in this section.

Walk the list once now, then use it as a review checklist before shipping any code that calls AWS, and again as a first response during any live incident.

How to Use This List

• Each item is a rule stated positively, with a one-line rationale.
• Groups run in roughly the order you'd hit them in a real incident: error handling first, then auth, retries, consistency, and observability/response.
• Check the boxes against your own code as a quick audit.
• Revisit whenever you add a new long-running process, a new retry loop, or a new client that touches a differently-consistent service.

A - Error Handling

• Branch on the error code, never the message. Use err.response["Error"]["Code"] (boto3) or error.name (SDK v3); message text is human-readable and can change.
• Always log the request id. Capture ResponseMetadata.RequestId (boto3) or $metadata.requestId (SDK v3) on every failure for tracing and AWS Support.
• Check status code alongside error code. 4xx usually means fix the request; 5xx is usually transient and already retried by the SDK.
• Re-raise anything unexpected. Swallow only the specific codes you understand and intend to handle.
• Sort every incident into a bucket fast. Auth, throttling, retries, or consistency - the bucket points directly at the right playbook.

Pull code, request id, and status together, in one place.

# --- Python (boto3) ---
from botocore.exceptions import ClientError
 
try:
    client.some_operation()
except ClientError as err:
    print({
        "code": err.response["Error"]["Code"],
        "request_id": err.response["ResponseMetadata"]["RequestId"],
        "status": err.response["ResponseMetadata"]["HTTPStatusCode"],
    })
    raise

// --- TypeScript (AWS SDK v3) ---
try {
  await client.send(command);
} catch (err) {
  const e = err as { name?: string; $metadata?: { requestId?: string; httpStatusCode?: number } };
  console.log({ code: e.name, requestId: e.$metadata?.requestId, status: e.$metadata?.httpStatusCode });
  throw err;
}

B - Credentials & Auth

• Never cache a raw credential snapshot. Build clients from a session, profile, or credential provider function so the SDK can refresh temporary credentials itself.
• Use fromTemporaryCredentials (SDK v3) or a role-assuming profile (boto3) instead of extracting AccessKeyId/SecretAccessKey/SessionToken into variables.
• Treat SSO sessions as expiring, not permanent. A long-running process that captured an SSO session at startup will fail once that session lapses.
• Alarm specifically on ExpiredTokenException/ExpiredToken. A dedicated alarm surfaces this bucket faster than a generic error-rate signal.
• Test long-running processes past a credential's TTL, not just in a short-lived integration test that never crosses expiry.

C - Retries & Backoff

• Rely on the SDK's built-in retry strategy. Configure botocore.config.Config(retries={...}) (boto3) or maxAttempts/retryStrategy (SDK v3) instead of writing a retry loop by hand.
• Never reimplement backoff yourself. A hand-rolled loop almost always drops jitter, which is what prevents synchronized retry storms across many clients.
• Use adaptive mode for hot, frequently throttled paths. It adds client-side rate estimation on top of backoff and jitter.
• Add a circuit breaker around dependencies with a history of degrading. Retries alone cannot stop a storm once one starts; stopping calls entirely can.
• Pair aggressive retry tuning with idempotency tokens on writes. Amplified retries against a non-idempotent create can duplicate its effect.

Configure retries on the client once - do not write the loop yourself.

# --- Python (boto3) ---
from botocore.config import Config
 
cfg = Config(retries={"max_attempts": 5, "mode": "adaptive"})
ddb = boto3.client("dynamodb", config=cfg)

// --- TypeScript (AWS SDK v3) ---
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { AdaptiveRetryStrategy } from "@aws-sdk/util-retry";
 
const ddb = new DynamoDBClient({ retryStrategy: new AdaptiveRetryStrategy(async () => 5) });

D - Consistency Assumptions

• Drop any S3 retry-after-write workaround. S3 has been strongly consistent for read-after-write since December 2020; the old pattern is unnecessary.
• Use ConsistentRead=True/ConsistentRead: true on DynamoDB reads that need read-your-own-write correctness, not on every read by default (it costs double capacity and isn't available on GSIs).
• Add a short, bounded retry the first time automation uses a newly created IAM role or policy. Global propagation can take a few seconds.
• Cap that IAM propagation retry. If AccessDenied persists past a handful of attempts, treat it as a real policy problem, not propagation.
• Know each service's specific model before assuming "eventual consistency" is the answer. It differs meaningfully by service and even by API within a service.

E - Monitoring & Incident Response

• Alarm on service-specific, bucket-specific metrics. DynamoDB's ThrottledRequests, specific auth error codes, and traffic correlation - not a single generic error-rate alarm.
• Follow a consistent five-stage playbook. Detect, triage, mitigate, root-cause, postmortem - every time, regardless of severity.
• Treat mitigation and root-cause fix as separate work. A rollback or restart resolves the symptom; track the underlying defect fix as its own action item.
• Write a blameless RCA with a consistent template. Timeline, impact, root cause, and specific, owned action items - framed around the system, not individuals.
• Correlate incident timing with your own traffic and deploys first. Most AWS SDK incidents are client-side in origin, not an AWS-side outage.

FAQs

Related

• Categories of AWS SDK Failures - the buckets these best practices are organized around.
• Credential Expiry & Silent Auth Failures - group B in depth.
• Retry Storms & Cascading Failure - group C in depth.
• Eventual Consistency Surprises - group D in depth.
• Production Incident Response for AWS-Backed Services - group E in depth.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).