Eventual Consistency Surprises

Q: Is DynamoDB always eventually consistent?

By default, yes, for GetItem, Query, and Scan. Pass ConsistentRead=True (boto3) or ConsistentRead: true (SDK v3) on a specific read to make it strongly consistent, at roughly double the read capacity cost.

Q: Can I use ConsistentRead on a Global Secondary Index?

No. GSIs only support eventually consistent reads; ConsistentRead=True is only valid against the base table or a Local Secondary Index.

Q: How do I tell propagation delay apart from a real IAM misconfiguration?

If AccessDenied clears within a few seconds and a handful of retries, it was propagation. If it persists well beyond that, treat it as an actual policy or trust-relationship problem and inspect the policy itself.

A common bug report reads: "I wrote the record, then read it back immediately, and it wasn't there." The fix depends entirely on which service you're talking about - and one of the most common assumptions engineers carry into this bug is now outdated.

This page walks through the three places this surprise shows up in an AWS SDK codebase - S3, DynamoDB, and IAM - and the actual, current consistency model for each.

Recipe

Before assuming "eventual consistency" explains a missing read, check whether the service you're using is even eventually consistent for that operation - it may not be.

# --- Python (boto3) ---
# S3: strongly consistent since December 2020. No retry-after-write needed.
s3.put_object(Bucket="my-bucket", Key="orders/o-1.json", Body=b"{}")
resp = s3.get_object(Bucket="my-bucket", Key="orders/o-1.json")  # always sees the write immediately

// --- TypeScript (AWS SDK v3) ---
// S3: strongly consistent since December 2020. No retry-after-write needed.
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
 
await s3.send(new PutObjectCommand({ Bucket: "my-bucket", Key: "orders/o-1.json", Body: "{}" }));
const resp = await s3.send(new GetObjectCommand({ Bucket: "my-bucket", Key: "orders/o-1.json" })); // sees it immediately

Working Example

Case 1: S3 - Not Actually Eventually Consistent Anymore

The outdated misconception: "S3 is eventually consistent, so I need to poll or add a delay after a write before reading it back."

This was true before December 2020. It is not true anymore. AWS made all S3 operations strongly consistent for read-after-write, including overwrite PUTs and DELETEs - a GET or LIST immediately following a successful write now reflects that write, with no special configuration and no retry-and-hope pattern required.

# --- Python (boto3) ---
# There is no "wait for consistency" step to add here - this is correct as written.
s3.put_object(Bucket="my-bucket", Key="config.json", Body=b'{"flag": true}')
obj = s3.get_object(Bucket="my-bucket", Key="config.json")
print(obj["Body"].read())  # reflects the write that just happened, every time

// --- TypeScript (AWS SDK v3) ---
// No retry-after-write loop needed here either.
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
 
await s3.send(new PutObjectCommand({ Bucket: "my-bucket", Key: "config.json", Body: '{"flag": true}' }));
const obj = await s3.send(new GetObjectCommand({ Bucket: "my-bucket", Key: "config.json" }));

If you inherited code with a retry-after-S3-write loop "just in case," it is very likely a holdover from before 2020 and can be safely removed - it was never wrong to have it, but it is no longer necessary.

Case 2: DynamoDB - Eventually Consistent Reads by Default

Unlike S3, DynamoDB's default read behavior (GetItem, Query, Scan) is genuinely eventually consistent - a read shortly after a write can, in rare cases, return the prior value. This is the trade-off DynamoDB makes by default for lower latency and higher throughput per read.

# --- Python (boto3) ---
ddb.put_item(TableName="orders", Item={"id": {"S": "o-1"}, "status": {"S": "paid"}})
 
# Default read: eventually consistent - can rarely return a stale value.
resp = ddb.get_item(TableName="orders", Key={"id": {"S": "o-1"}})
 
# Strongly consistent read: guarantees the latest write is reflected.
resp_strong = ddb.get_item(
    TableName="orders",
    Key={"id": {"S": "o-1"}},
    ConsistentRead=True,
)

// --- TypeScript (AWS SDK v3) ---
import { PutItemCommand, GetItemCommand } from "@aws-sdk/client-dynamodb";
 
await ddb.send(new PutItemCommand({ TableName: "orders", Item: { id: { S: "o-1" }, status: { S: "paid" } } }));
 
// Default read: eventually consistent.
const resp = await ddb.send(new GetItemCommand({ TableName: "orders", Key: { id: { S: "o-1" } } }));
 
// Strongly consistent read: guarantees the latest write is reflected.
const respStrong = await ddb.send(
  new GetItemCommand({ TableName: "orders", Key: { id: { S: "o-1" } }, ConsistentRead: true }),
);

ConsistentRead: true costs twice the read capacity of an eventually consistent read and is not available on global secondary indexes at all - use it deliberately on the specific reads where correctness after a write actually matters (like a read-your-own-write flow right after a user submits a form), not on every read by default.

Case 3: IAM - Global Propagation Delay

IAM is a different kind of consistency surprise: it is not about reading stale data, it is about a brand-new or just-updated role/policy not yet being honored everywhere. IAM is a global service, and changes need a short window - typically a few seconds, occasionally longer - to propagate across AWS's infrastructure.

# --- Python (boto3) ---
import time
from botocore.exceptions import ClientError
 
iam.create_role(RoleName="new-worker-role", AssumeRolePolicyDocument="...")
iam.attach_role_policy(RoleName="new-worker-role", PolicyArn="arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess")
 
# A call using this role immediately after creation can still fail with
# AccessDenied while the change propagates - a short bounded retry absorbs it.
for attempt in range(5):
    try:
        sts.assume_role(RoleArn="arn:aws:iam::123456789012:role/new-worker-role", RoleSessionName="probe")
        break
    except ClientError as err:
        if err.response["Error"]["Code"] == "AccessDenied" and attempt < 4:
            time.sleep(2 ** attempt)  # short, bounded backoff for propagation only
        else:
            raise

// --- TypeScript (AWS SDK v3) ---
import { CreateRoleCommand, AttachRolePolicyCommand } from "@aws-sdk/client-iam";
import { AssumeRoleCommand } from "@aws-sdk/client-sts";
 
await iam.send(new CreateRoleCommand({ RoleName: "new-worker-role", AssumeRolePolicyDocument: "..." }));
await iam.send(
  new AttachRolePolicyCommand({ RoleName: "new-worker-role", PolicyArn: "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess" }),
);
 
for (let attempt = 0; attempt < 5; attempt++) {
  try {
    await sts.send(
      new AssumeRoleCommand({ RoleArn: "arn:aws:iam::123456789012:role/new-worker-role", RoleSessionName: "probe" }),
    );
    break;
  } catch (err) {
    const name = (err as { name?: string }).name;
    if (name === "AccessDenied" && attempt < 4) {
      await new Promise((r) => setTimeout(r, 2 ** attempt * 1000)); // short, bounded backoff
    } else {
      throw err;
    }
  }
}

This is why infrastructure automation that creates a role and immediately uses it in the same script often needs a short, bounded wait or retry - not because IAM is unreliable, but because global propagation genuinely takes a moment.

Gotchas

Assuming S3 still needs a retry-after-write pattern - this was true before December 2020 and is an outdated, common misconception now. Fix: remove old S3 consistency workarounds; a GET right after a PUT is guaranteed current.
Using DynamoDB's default read where correctness matters - a read-your-own-write flow (like showing a user their just-submitted order) can show stale data with eventually consistent reads. Fix: pass ConsistentRead=True on the specific reads that need it.
Using ConsistentRead everywhere by default - it doubles read capacity cost and isn't available on GSIs. Fix: reserve it for reads that specifically need read-your-own-write correctness.
Using a brand-new IAM role immediately in automation - propagation delay can cause a transient AccessDenied. Fix: add a short, bounded retry specifically around the first use of a newly created role or policy.
Confusing IAM propagation delay with a real permissions bug - if the AccessDenied persists past a few tries, it is very likely an actual policy problem, not propagation. Fix: cap the bounded retry and treat a still-failing call after that as a real permissions issue to fix.

Alternatives

Alternative	Use When	Don't Use When
S3 default consistency (no workaround)	Any S3 read-after-write scenario	Never needed as a workaround anymore - remove legacy retry-after-S3-write code
DynamoDB `ConsistentRead=True`	A specific read must reflect the immediately preceding write	High-throughput reads where eventual consistency is acceptable and cost/capacity matters
DynamoDB transactions (`TransactWriteItems`)	Multiple items must be consistent with each other atomically	A single-item read/write where `ConsistentRead` already solves the problem
Bounded retry after IAM changes	Automation that creates and immediately uses a role or policy	Long-running services where the role already existed well before use

FAQs

Do I still need to worry about S3 eventual consistency?

No, not for read-after-write. AWS made all S3 operations strongly consistent in December 2020. Any code that adds delays or retries specifically to work around S3 consistency is solving a problem that no longer exists.

Is DynamoDB always eventually consistent?

By default, yes, for GetItem, Query, and Scan. Pass ConsistentRead=True (boto3) or ConsistentRead: true (SDK v3) on a specific read to make it strongly consistent, at roughly double the read capacity cost.

Can I use ConsistentRead on a Global Secondary Index?

No. GSIs only support eventually consistent reads; ConsistentRead=True is only valid against the base table or a Local Secondary Index.

How long does IAM propagation actually take?

Typically a few seconds, though AWS does not guarantee an exact bound. A short, bounded retry (a handful of attempts with backoff) comfortably absorbs the typical window.

How do I tell propagation delay apart from a real IAM misconfiguration?

If AccessDenied clears within a few seconds and a handful of retries, it was propagation. If it persists well beyond that, treat it as an actual policy or trust-relationship problem and inspect the policy itself.

Categories of AWS SDK Failures - where consistency surprises fit among the other failure buckets.
Debugging Basics - reading the error code before assuming the cause.
Credential Expiry & Silent Auth Failures - a different kind of "AccessDenied" surprise.
Debugging & Incident Response Best Practices - habits that prevent chasing outdated consistency assumptions.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

This page walks through the three places this surprise shows up in an AWS SDK codebase - S3, DynamoDB, and IAM - and the actual, current consistency model for each.

Recipe

Before assuming "eventual consistency" explains a missing read, check whether the service you're using is even eventually consistent for that operation - it may not be.

# --- Python (boto3) ---
# S3: strongly consistent since December 2020. No retry-after-write needed.
s3.put_object(Bucket="my-bucket", Key="orders/o-1.json", Body=b"{}")
resp = s3.get_object(Bucket="my-bucket", Key="orders/o-1.json")  # always sees the write immediately

// --- TypeScript (AWS SDK v3) ---
// S3: strongly consistent since December 2020. No retry-after-write needed.
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
 
await s3.send(new PutObjectCommand({ Bucket: "my-bucket", Key: "orders/o-1.json", Body: "{}" }));
const resp = await s3.send(new GetObjectCommand({ Bucket: "my-bucket", Key: "orders/o-1.json" })); // sees it immediately

Working Example

Case 1: S3 - Not Actually Eventually Consistent Anymore

The outdated misconception: "S3 is eventually consistent, so I need to poll or add a delay after a write before reading it back."

# --- Python (boto3) ---
# There is no "wait for consistency" step to add here - this is correct as written.
s3.put_object(Bucket="my-bucket", Key="config.json", Body=b'{"flag": true}')
obj = s3.get_object(Bucket="my-bucket", Key="config.json")
print(obj["Body"].read())  # reflects the write that just happened, every time

// --- TypeScript (AWS SDK v3) ---
// No retry-after-write loop needed here either.
import { PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
 
await s3.send(new PutObjectCommand({ Bucket: "my-bucket", Key: "config.json", Body: '{"flag": true}' }));
const obj = await s3.send(new GetObjectCommand({ Bucket: "my-bucket", Key: "config.json" }));

Case 2: DynamoDB - Eventually Consistent Reads by Default

# --- Python (boto3) ---
ddb.put_item(TableName="orders", Item={"id": {"S": "o-1"}, "status": {"S": "paid"}})
 
# Default read: eventually consistent - can rarely return a stale value.
resp = ddb.get_item(TableName="orders", Key={"id": {"S": "o-1"}})
 
# Strongly consistent read: guarantees the latest write is reflected.
resp_strong = ddb.get_item(
    TableName="orders",
    Key={"id": {"S": "o-1"}},
    ConsistentRead=True,
)

// --- TypeScript (AWS SDK v3) ---
import { PutItemCommand, GetItemCommand } from "@aws-sdk/client-dynamodb";
 
await ddb.send(new PutItemCommand({ TableName: "orders", Item: { id: { S: "o-1" }, status: { S: "paid" } } }));
 
// Default read: eventually consistent.
const resp = await ddb.send(new GetItemCommand({ TableName: "orders", Key: { id: { S: "o-1" } } }));
 
// Strongly consistent read: guarantees the latest write is reflected.
const respStrong = await ddb.send(
  new GetItemCommand({ TableName: "orders", Key: { id: { S: "o-1" } }, ConsistentRead: true }),
);

Case 3: IAM - Global Propagation Delay

# --- Python (boto3) ---
import time
from botocore.exceptions import ClientError
 
iam.create_role(RoleName="new-worker-role", AssumeRolePolicyDocument="...")
iam.attach_role_policy(RoleName="new-worker-role", PolicyArn="arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess")
 
# A call using this role immediately after creation can still fail with
# AccessDenied while the change propagates - a short bounded retry absorbs it.
for attempt in range(5):
    try:
        sts.assume_role(RoleArn="arn:aws:iam::123456789012:role/new-worker-role", RoleSessionName="probe")
        break
    except ClientError as err:
        if err.response["Error"]["Code"] == "AccessDenied" and attempt < 4:
            time.sleep(2 ** attempt)  # short, bounded backoff for propagation only
        else:
            raise

// --- TypeScript (AWS SDK v3) ---
import { CreateRoleCommand, AttachRolePolicyCommand } from "@aws-sdk/client-iam";
import { AssumeRoleCommand } from "@aws-sdk/client-sts";
 
await iam.send(new CreateRoleCommand({ RoleName: "new-worker-role", AssumeRolePolicyDocument: "..." }));
await iam.send(
  new AttachRolePolicyCommand({ RoleName: "new-worker-role", PolicyArn: "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess" }),
);
 
for (let attempt = 0; attempt < 5; attempt++) {
  try {
    await sts.send(
      new AssumeRoleCommand({ RoleArn: "arn:aws:iam::123456789012:role/new-worker-role", RoleSessionName: "probe" }),
    );
    break;
  } catch (err) {
    const name = (err as { name?: string }).name;
    if (name === "AccessDenied" && attempt < 4) {
      await new Promise((r) => setTimeout(r, 2 ** attempt * 1000)); // short, bounded backoff
    } else {
      throw err;
    }
  }
}

Gotchas

Assuming S3 still needs a retry-after-write pattern - this was true before December 2020 and is an outdated, common misconception now. Fix: remove old S3 consistency workarounds; a GET right after a PUT is guaranteed current.
Using DynamoDB's default read where correctness matters - a read-your-own-write flow (like showing a user their just-submitted order) can show stale data with eventually consistent reads. Fix: pass ConsistentRead=True on the specific reads that need it.
Using ConsistentRead everywhere by default - it doubles read capacity cost and isn't available on GSIs. Fix: reserve it for reads that specifically need read-your-own-write correctness.
Using a brand-new IAM role immediately in automation - propagation delay can cause a transient AccessDenied. Fix: add a short, bounded retry specifically around the first use of a newly created role or policy.
Confusing IAM propagation delay with a real permissions bug - if the AccessDenied persists past a few tries, it is very likely an actual policy problem, not propagation. Fix: cap the bounded retry and treat a still-failing call after that as a real permissions issue to fix.

Alternatives

Alternative	Use When	Don't Use When
S3 default consistency (no workaround)	Any S3 read-after-write scenario	Never needed as a workaround anymore - remove legacy retry-after-S3-write code
DynamoDB `ConsistentRead=True`	A specific read must reflect the immediately preceding write	High-throughput reads where eventual consistency is acceptable and cost/capacity matters
DynamoDB transactions (`TransactWriteItems`)	Multiple items must be consistent with each other atomically	A single-item read/write where `ConsistentRead` already solves the problem
Bounded retry after IAM changes	Automation that creates and immediately uses a role or policy	Long-running services where the role already existed well before use

FAQs

Do I still need to worry about S3 eventual consistency?

Is DynamoDB always eventually consistent?

Can I use ConsistentRead on a Global Secondary Index?

No. GSIs only support eventually consistent reads; ConsistentRead=True is only valid against the base table or a Local Secondary Index.

How long does IAM propagation actually take?

Typically a few seconds, though AWS does not guarantee an exact bound. A short, bounded retry (a handful of attempts with backoff) comfortably absorbs the typical window.

How do I tell propagation delay apart from a real IAM misconfiguration?

Categories of AWS SDK Failures - where consistency surprises fit among the other failure buckets.
Debugging Basics - reading the error code before assuming the cause.
Credential Expiry & Silent Auth Failures - a different kind of "AccessDenied" surprise.
Debugging & Incident Response Best Practices - habits that prevent chasing outdated consistency assumptions.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Eventual Consistency Surprises

Search Documentation

Eventual Consistency Surprises