Architecture & Resilience Best Practices

These are the habits that keep an AWS-backed service resilient as it grows, without requiring a rewrite every time traffic or scope changes.

Walk the list once now, then treat it as a review checklist whenever you design a new integration or revisit an existing one.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run roughly in order of where a failure first shows up: failure modes first, then isolation, safety, account structure, and long-term evolution.
Check the boxes against your own service as a quick architecture audit.
Revisit when you add a new downstream dependency, cross an account boundary, or scale up meaningfully.

A - Failure Modes as Design Input

Design for throttling, not around it. Let the SDK's built-in retry and backoff absorb transient rate limits instead of treating every throttle as an outage.
Account for eventual consistency explicitly. Use a strongly consistent read where the API supports it for read-after-write paths; otherwise design the workflow to tolerate a brief staleness window.
Always check per-item results on batch calls. BatchWriteItem, SendMessageBatch, and similar can return partial success in an otherwise-successful response.
Do not assume one downstream call's failure is isolated. A fan-out to several AWS calls has as many independent chances to fail as calls made.

Check the response's per-item results, not just the overall status.

# --- Python (boto3) ---
resp = sqs.send_message_batch(QueueUrl=queue_url, Entries=entries)
for failed in resp.get("Failed", []):
    print("failed:", failed["Id"], failed["Code"])  # re-drive these

// --- TypeScript (AWS SDK v3) ---
const resp = await sqs.send(new SendMessageBatchCommand({ QueueUrl: queueUrl, Entries: entries }));
for (const failed of resp.Failed ?? []) {
  console.log("failed:", failed.Id, failed.Code); // re-drive these
}

B - Isolating Failing Dependencies

Wrap unreliable dependencies in a circuit breaker. Fail fast after a failure threshold instead of piling up timeouts against a struggling dependency.
Give each dependency its own bounded pool. A per-dependency thread pool or concurrency limiter (a bulkhead) stops one slow dependency from starving another's capacity.
Do not use a circuit breaker as a substitute for retry configuration. Keep the SDK's retry/backoff settings; layer the breaker on top for sustained failures.
Tune the breaker's reset window to real recovery time. Too short causes flapping; too long delays recovery once the dependency is healthy again.

C - Idempotent, Retry-Safe Operations

Give every write operation a natural idempotency key. An order id, client-supplied key, or event id - never a value regenerated per retry attempt.
Guard with an atomic conditional write, not a check-then-act. A DynamoDB attribute_not_exists condition is race-free; a separate read-then-write is not.
Treat a conditional-check failure as a success, not an error. It usually means the operation already ran - the correct outcome for an idempotent design.
Expire idempotency guard records with a TTL. An unbounded ledger table grows forever if nothing ever removes old entries.

D - Multi-Account Structure

Split workloads across accounts by environment or team. Account boundaries are a stronger blast-radius control than IAM policy alone within a single account.
Route cross-account access through AssumeRole, never shared credentials. Every hop should go through an explicit trust relationship with its own scoped permissions.
Treat target account and role as configuration. Do not hardcode account ids or role ARNs into client construction logic.
Use AWS Organizations and Service Control Policies as guardrails. SCPs cap what any identity in an account can do, on top of - not instead of - IAM.

E - Documenting and Evolving Decisions

Write an ADR for decisions that are expensive to reverse. Service choice, region strategy, and account structure all qualify; routine implementation details usually do not.
Never edit an accepted ADR - supersede it. Write a new ADR and link back, preserving the history of why the architecture changed.
Apply the Well-Architected pillars at the code level, not just infrastructure. Structured logging, least privilege, retries/idempotency, client reuse, and pagination/batching are all code-level pillar work.
Evolve incrementally, not through a big-bang rewrite. Introduce a circuit breaker, a bulkhead, or an idempotency guard around one dependency at a time as it proves it needs one, rather than redesigning the whole service at once.

FAQs

Where should I start if my service has none of this?

Start with Group A - handle throttling, eventual consistency, and partial failure explicitly - since the rest of the section builds on treating those as normal, expected conditions rather than edge cases.

Do I need a circuit breaker for every dependency?

No. Reserve breakers for dependencies whose sustained failure meaningfully affects your service. A rarely-called, low-stakes dependency does not need one.

Is idempotent design only relevant if I retry manually?

No. At-least-once triggers like SQS standard queues, and any circuit breaker letting calls back through after reopening, can redeliver or retry without your code explicitly asking for it - so idempotent design matters even for "automatic" retries.

When should I split into multiple AWS accounts?

When a mistake or compromise in one environment or team's resources should not be able to reach another's. Small prototypes can reasonably stay in one account; production workloads handling real data or spend usually should not.

How do I know when a decision is worth an ADR?

If reversing it later would be expensive - in migration effort, cost, or risk - it is worth recording. Routine implementation choices that are easy to change later usually are not.

What does "evolving incrementally" mean in practice?

Adding resilience patterns (a breaker, a bulkhead, an idempotency guard) around one dependency or operation at a time, driven by evidence it needs one, rather than redesigning the whole service's architecture in a single big-bang effort.

How often should I revisit this checklist?

Whenever you add a new downstream dependency, cross a new account boundary, or the service's traffic or scope changes meaningfully - a pattern that was fine at low volume can violate several of these practices once it grows.

Does this list replace the individual pages in this section?

No, it condenses them into a review checklist. Each item links conceptually back to a page - Designing for AWS's Failure Modes, Circuit Breakers & Bulkheads, and the rest - for the full reasoning and code.

Designing for AWS's Failure Modes - the failure shapes behind Group A.
Circuit Breakers & Bulkheads Around AWS Calls - the isolation patterns behind Group B.
Idempotent Service Design Around SDK Calls - the retry-safety patterns behind Group C.
Multi-Account Architecture for AWS SDK Applications - the account structure behind Group D.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Architecture & Resilience Best Practices

These are the habits that keep an AWS-backed service resilient as it grows, without requiring a rewrite every time traffic or scope changes.

Walk the list once now, then treat it as a review checklist whenever you design a new integration or revisit an existing one.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run roughly in order of where a failure first shows up: failure modes first, then isolation, safety, account structure, and long-term evolution.
Check the boxes against your own service as a quick architecture audit.
Revisit when you add a new downstream dependency, cross an account boundary, or scale up meaningfully.

A - Failure Modes as Design Input

Design for throttling, not around it. Let the SDK's built-in retry and backoff absorb transient rate limits instead of treating every throttle as an outage.
Account for eventual consistency explicitly. Use a strongly consistent read where the API supports it for read-after-write paths; otherwise design the workflow to tolerate a brief staleness window.
Always check per-item results on batch calls. BatchWriteItem, SendMessageBatch, and similar can return partial success in an otherwise-successful response.
Do not assume one downstream call's failure is isolated. A fan-out to several AWS calls has as many independent chances to fail as calls made.

Check the response's per-item results, not just the overall status.

# --- Python (boto3) ---
resp = sqs.send_message_batch(QueueUrl=queue_url, Entries=entries)
for failed in resp.get("Failed", []):
    print("failed:", failed["Id"], failed["Code"])  # re-drive these

// --- TypeScript (AWS SDK v3) ---
const resp = await sqs.send(new SendMessageBatchCommand({ QueueUrl: queueUrl, Entries: entries }));
for (const failed of resp.Failed ?? []) {
  console.log("failed:", failed.Id, failed.Code); // re-drive these
}

B - Isolating Failing Dependencies

Wrap unreliable dependencies in a circuit breaker. Fail fast after a failure threshold instead of piling up timeouts against a struggling dependency.
Give each dependency its own bounded pool. A per-dependency thread pool or concurrency limiter (a bulkhead) stops one slow dependency from starving another's capacity.
Do not use a circuit breaker as a substitute for retry configuration. Keep the SDK's retry/backoff settings; layer the breaker on top for sustained failures.
Tune the breaker's reset window to real recovery time. Too short causes flapping; too long delays recovery once the dependency is healthy again.

C - Idempotent, Retry-Safe Operations

Give every write operation a natural idempotency key. An order id, client-supplied key, or event id - never a value regenerated per retry attempt.
Guard with an atomic conditional write, not a check-then-act. A DynamoDB attribute_not_exists condition is race-free; a separate read-then-write is not.
Treat a conditional-check failure as a success, not an error. It usually means the operation already ran - the correct outcome for an idempotent design.
Expire idempotency guard records with a TTL. An unbounded ledger table grows forever if nothing ever removes old entries.

D - Multi-Account Structure

Split workloads across accounts by environment or team. Account boundaries are a stronger blast-radius control than IAM policy alone within a single account.
Route cross-account access through AssumeRole, never shared credentials. Every hop should go through an explicit trust relationship with its own scoped permissions.
Treat target account and role as configuration. Do not hardcode account ids or role ARNs into client construction logic.
Use AWS Organizations and Service Control Policies as guardrails. SCPs cap what any identity in an account can do, on top of - not instead of - IAM.

E - Documenting and Evolving Decisions

Write an ADR for decisions that are expensive to reverse. Service choice, region strategy, and account structure all qualify; routine implementation details usually do not.
Never edit an accepted ADR - supersede it. Write a new ADR and link back, preserving the history of why the architecture changed.
Apply the Well-Architected pillars at the code level, not just infrastructure. Structured logging, least privilege, retries/idempotency, client reuse, and pagination/batching are all code-level pillar work.
Evolve incrementally, not through a big-bang rewrite. Introduce a circuit breaker, a bulkhead, or an idempotency guard around one dependency at a time as it proves it needs one, rather than redesigning the whole service at once.

FAQs

Where should I start if my service has none of this?

Do I need a circuit breaker for every dependency?

No. Reserve breakers for dependencies whose sustained failure meaningfully affects your service. A rarely-called, low-stakes dependency does not need one.

Is idempotent design only relevant if I retry manually?

When should I split into multiple AWS accounts?

How do I know when a decision is worth an ADR?

If reversing it later would be expensive - in migration effort, cost, or risk - it is worth recording. Routine implementation choices that are easy to change later usually are not.

What does "evolving incrementally" mean in practice?

How often should I revisit this checklist?

Does this list replace the individual pages in this section?

Designing for AWS's Failure Modes - the failure shapes behind Group A.
Circuit Breakers & Bulkheads Around AWS Calls - the isolation patterns behind Group B.
Idempotent Service Design Around SDK Calls - the retry-safety patterns behind Group C.
Multi-Account Architecture for AWS SDK Applications - the account structure behind Group D.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Architecture & Resilience Best Practices

Buscar en la documentación

Architecture & Resilience Best Practices