Designing for AWS's Failure Modes

Every AWS SDK call can fail in ways that have nothing to do with your code being wrong.

A well-formed request can be throttled because your account or table is busy. A write can succeed but not yet be visible to a read a few milliseconds later. A batch operation touching ten items can report seven successes and three failures in the same response. None of these are bugs. They are the normal, documented behavior of a distributed system, and treating them as first-class design inputs - not edge cases you patch in after an incident - is the foundation of the rest of this section.

Summary

• AWS's three recurring failure shapes are throttling, eventual consistency, and partial (multi-resource) failure.
• Insight: these are not exceptions to a reliable system, they are the normal operating behavior of any system built at AWS's scale.
• Key Concepts: throttling, eventual consistency, partial failure, idempotency, graceful degradation.
• When to Use: any service that calls AWS APIs in a request path or a background job, from a single Lambda to a multi-service architecture.
• Limitations/Trade-offs: designing for these modes adds code (retries, reconciliation, per-item error handling) that a purely happy-path implementation skips.
• Related Topics: exponential backoff and jitter, structured error handling, circuit breakers and bulkheads, idempotent service design.

Foundations

Throttling

AWS services protect themselves and their other tenants with rate limits. When you exceed one, the API returns a throttling error - ThrottlingException, ProvisionedThroughputExceededException, TooManyRequestsException, depending on the service - instead of queuing your request. This is not a failure of the service; it is the service telling you to slow down. Both boto3 and the AWS SDK for JavaScript v3 retry throttling errors automatically with exponential backoff, covered in depth in Exponential Backoff & Jitter Configuration. Your job at the architecture level is to make sure the caller of your service can tolerate the added latency of a retried call, and to avoid retry storms where many clients back off and retry in lockstep.

Eventual Consistency

Some AWS data stores and indexes are eventually consistent by default. A DynamoDB Global Secondary Index, an S3 list operation after a delete, and cross-region replication all have a window - typically milliseconds to seconds - during which a write is durable but not yet visible everywhere a read might look. This is a deliberate trade-off for availability and throughput, not a defect. Code that writes an item and immediately reads it back through a path that is not strongly consistent can observe stale or missing data, and needs to either request strong consistency where the API supports it, or be written to tolerate the lag.

Partial Failure

Multi-item and multi-resource operations - BatchWriteItem, PutRecords on Kinesis, SendMessageBatch on SQS - do not fail all-or-nothing. They return a per-item result: some entries succeed, some fail, and the response tells you which is which. Code that checks only the overall HTTP status and assumes success silently drops the failed items.

# --- Python (boto3) ---
import boto3
 
sqs = boto3.client("sqs", region_name="us-east-1")
resp = sqs.send_message_batch(
    QueueUrl="https://sqs.us-east-1.amazonaws.com/111122223333/orders",
    Entries=[
        {"Id": "1", "MessageBody": "order-1"},
        {"Id": "2", "MessageBody": "order-2"},
    ],
)
 
# A 200 response can still contain per-message failures.
for failed in resp.get("Failed", []):
    print("failed:", failed["Id"], failed["Code"])

// --- TypeScript (AWS SDK v3) ---
import { SQSClient, SendMessageBatchCommand } from "@aws-sdk/client-sqs";
 
const sqs = new SQSClient({ region: "us-east-1" });
const resp = await sqs.send(new SendMessageBatchCommand({
  QueueUrl: "https://sqs.us-east-1.amazonaws.com/111122223333/orders",
  Entries: [
    { Id: "1", MessageBody: "order-1" },
    { Id: "2", MessageBody: "order-2" },
  ],
}));
 
// A 200 response can still contain per-message failures.
for (const failed of resp.Failed ?? []) {
  console.log("failed:", failed.Id, failed.Code);
}

Mechanics & Interactions

These three modes interact with each other more than they interact in isolation. A throttled batch write can leave you with a partial failure that itself needs to be retried, and the retried subset can then race with an eventually consistent read from a different part of your system that already assumed the whole batch landed. Designing for one in isolation and ignoring the others is a common source of production incidents.

The practical response has a consistent shape across services. For throttling, let the SDK's built-in retry and backoff do the work, and design your call volume so retries are the exception rather than the steady state. For eventual consistency, use a strongly consistent read where the API offers one (DynamoDB's ConsistentRead, for example) when you truly need read-after-write, and otherwise design your workflow so a stale read is harmless - an eventually-settling counter, a cache that self-corrects. For partial failure, always inspect the per-item result of a batch call and re-drive the failed subset, rather than trusting the overall call status.

These patterns compose into the more structured tools covered elsewhere in this section: a circuit breaker stops you from hammering a dependency that is failing outright, and idempotent service design makes it safe to retry the partial failures and throttled calls that this page describes.

Advanced Considerations & Applications

At larger scale, these failure modes compound with your own architecture choices. A service that fans out one incoming request into ten downstream AWS calls has ten independent chances to be throttled, ten independent consistency windows, and ten independent partial-failure surfaces - and the overall request is only as reliable as the weakest of those calls unless you design deliberately for graceful degradation. Graceful degradation means deciding, ahead of time, which downstream failures are fatal to the overall request and which can be skipped, defaulted, or retried asynchronously without blocking the caller.

Multi-region and multi-account architectures add another layer: cross-region replication (S3 Cross-Region Replication, DynamoDB Global Tables) is eventually consistent by definition, and a failover plan that assumes instant consistency between regions will surprise you during an actual regional event. Bake in an explicit, measured expectation for replication lag rather than an implicit assumption of zero lag.

The Well-Architected Framework's Reliability pillar formalizes much of this thinking - designing to withstand component failure, testing recovery procedures, and automating recovery where possible - and is covered from a code-level angle in Well-Architected Framework Through an SDK Lens.

Common Misconceptions

• "A throttling error means AWS is down." - No, it means you exceeded a rate limit; the service is healthy and the SDK's default retry usually resolves it transparently.
• "If my write call returned 200, the data is everywhere it needs to be." - Not necessarily; eventually consistent indexes and cross-region replicas can lag behind a successful write.
• "A batch API either fully succeeds or fully fails." - No, batch operations commonly return mixed per-item results in a single successful HTTP response.
• "Retrying always fixes a failed call." - Only if the underlying cause is transient and the operation is safe to retry; see idempotent service design for the caveats.
• "These failure modes only matter at very large scale." - They show up in small systems too, just less frequently - which makes them easier to miss during testing.

FAQs

Related

• Architecture Basics - a minimal resilient-call wrapper that responds to these failure modes.
• Circuit Breakers & Bulkheads Around AWS Calls - isolating and failing fast against a struggling dependency.
• Idempotent Service Design Around SDK Calls - making retries of throttled and partially failed calls safe.
• Exponential Backoff & Jitter Configuration - the retry mechanics referenced throughout this page.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).