SNS & SQS via SDK Best Practices

This is the checklist to run against any messaging system built on SNS and SQS, beyond the Core intro's single topic and queue.

Each item is a rule stated positively, with a one-line rationale. Work top to bottom - fan-out and permissions first, then ordering, failure handling, throughput, and filtering.

How to Use This List

Treat each box as an audit question against your own messaging design.
The groups run roughly in the order you would build a fan-out system: wiring, ordering, failure handling, throughput, filtering.
Most rules apply whether messages carry business events, notifications, or background jobs.
Revisit the list whenever you add a new subscriber or a new event type to an existing topic.

Grant SNS explicit permission on every subscribed queue. Add a queue policy allowing sns.amazonaws.com to SendMessage, scoped with aws:SourceArn to the topic.
Subscribe with the queue's ARN, not its URL. Subscribe needs the ARN as the endpoint; the URL is only for SendMessage/ReceiveMessage.
Verify SQS subscriptions confirm automatically. Unlike email/HTTPS, they need no manual confirmation step, so a stuck PendingConfirmation means you subscribed the wrong protocol.
Let each subscriber drain independently. A slow or failing consumer on one queue must never block delivery to the others.

B - Ordering & Deduplication (FIFO)

Reach for FIFO only when order or exactly-once-ish delivery matters. It trades throughput for guarantees; standard is faster when order is not a requirement.
Name and configure FIFO resources at creation time. The .fifo suffix and FifoTopic/FifoQueue attribute cannot be added or removed after the fact.
Pick a MessageGroupId granular enough to parallelize. One group for all traffic serializes every consumer; group by customer or order instead.
Prefer ContentBasedDeduplication over hand-rolled dedup ids. Let SQS hash the body rather than tracking your own MessageDeduplicationId values.
Match FIFO topics to FIFO queues only. A FIFO topic rejects standard queues, email, and HTTPS subscribers outright.

# --- Python (boto3) ---
# FIFO naming and attribute are both required, set once at creation.
queue_url = sqs.create_queue(
    QueueName="orders.fifo",
    Attributes={"FifoQueue": "true", "ContentBasedDeduplication": "true"},
)["QueueUrl"]

// --- TypeScript (AWS SDK v3) ---
// FIFO naming and attribute are both required, set once at creation.
const { QueueUrl } = await sqs.send(new CreateQueueCommand({
  QueueName: "orders.fifo",
  Attributes: { FifoQueue: "true", ContentBasedDeduplication: "true" },
}));

C - Failure Handling (Dead-Letter Queues)

Attach a redrive policy to every production queue. No queue should retry a poison message forever with no escape hatch.
Set maxReceiveCount above 1. Allow a handful of retries (commonly 3-5) so transient failures do not permanently exile recoverable messages.
Match FIFO DLQs to FIFO source queues. A FIFO queue cannot target a standard queue as its dead-letter queue.
Alarm on DLQ depth. A CloudWatch alarm on ApproximateNumberOfMessagesVisible catches silent pileups before they go unnoticed for days.
Redrive with StartMessageMoveTask, not manual resends. Bulk-move messages back to the source queue once the underlying bug is fixed.

D - Throughput & Cost (Batching, Long Polling, Visibility)

Batch sends and deletes up to 10 per call. SendMessageBatch/DeleteMessageBatch cut request count (and cost) versus one-at-a-time calls.
Always enable long polling. Set WaitTimeSeconds (up to 20) so receives wait for messages instead of returning empty and billing anyway.
Size visibility timeouts above worst-case processing time. Or extend a specific message's timeout with ChangeMessageVisibility instead of inflating the default for everyone.
Check batch responses for partial failures. A batch call can partially succeed; always inspect Failed alongside Successful.
Make consumers idempotent. Standard queues are at-least-once, so duplicate delivery is expected behavior, not an edge case.

E - Selective Delivery (Filtering)

Filter at the subscription, not by splitting topics. One topic with per-subscriber FilterPolicy rules scales better than a topic per event type.
Always send the attributes your filters check. A filter on eventType matches nothing if publishers never set that attribute.
Match FilterPolicyScope to how the policy is written. MessageAttributes (default) and MessageBody are not interchangeable - set the scope explicitly if you filter on the body.
Treat non-matching deliveries as silent, not erroneous. Publish succeeds regardless of whether any given subscriber's filter matches.
Review filters periodically for drift. An overly broad filter quietly re-creates full unfiltered fan-out load.

FAQs

Where should I start when building a new fan-out system?

With permissions and wiring: get the queue policy and ARN-based subscription right first. A correctly wired fan-out with no ordering, filtering, or DLQ is still functional; a misconfigured queue policy silently drops every delivery.

When is FIFO worth the throughput trade-off?

When downstream logic genuinely depends on event order for a given entity, or when duplicate sends from a retrying producer would corrupt state. If neither applies, standard topics and queues are simpler and faster.

What is the single most common messaging mistake?

No dead-letter queue. Without one, a poison message retries indefinitely, burning consumer capacity, or a low maxReceiveCount discards messages that would have succeeded on retry.

How do I avoid processing the same message twice?

Delete only after successful processing using the correct receipt handle, size the visibility timeout above your processing time, and write idempotent consumer logic regardless - standard queues are at-least-once by design.

Should I always enable long polling?

Yes. There is essentially no downside: it reduces both empty responses and delivery latency compared to short polling.

Do filter policies replace the need for multiple topics?

In most cases yes. A single topic with per-subscription filter policies handles routing subsets of events far more flexibly than maintaining a topic per event type.

Why did my SNS-to-SQS fan-out silently fail?

The most common cause is a missing queue policy. SNS needs explicit sqs:SendMessage permission on each subscribed queue; without it, Publish still succeeds but delivery to that queue never happens.

Do these rules apply equally to FIFO and standard resources?

Most do. Batching, long polling, visibility tuning, dead-letter queues, and filtering all apply to FIFO resources too, with the added constraint that FIFO must pair with FIFO throughout (topic, queue, and DLQ).

SNS & SQS Together: Fan-Out Messaging - the wiring habits in group A.
FIFO Topics & Queues - the ordering habits in group B.
Dead-Letter Queues & Redrive Policies - the failure-handling habits in group C.
Batching, Long Polling & Visibility Timeouts - the throughput habits in group D.
Message Filtering with SNS Subscription Filters - the filtering habits in group E.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

SNS & SQS via SDK Best Practices

This is the checklist to run against any messaging system built on SNS and SQS, beyond the Core intro's single topic and queue.

Each item is a rule stated positively, with a one-line rationale. Work top to bottom - fan-out and permissions first, then ordering, failure handling, throughput, and filtering.

How to Use This List

Treat each box as an audit question against your own messaging design.
The groups run roughly in the order you would build a fan-out system: wiring, ordering, failure handling, throughput, filtering.
Most rules apply whether messages carry business events, notifications, or background jobs.
Revisit the list whenever you add a new subscriber or a new event type to an existing topic.

Grant SNS explicit permission on every subscribed queue. Add a queue policy allowing sns.amazonaws.com to SendMessage, scoped with aws:SourceArn to the topic.
Subscribe with the queue's ARN, not its URL. Subscribe needs the ARN as the endpoint; the URL is only for SendMessage/ReceiveMessage.
Verify SQS subscriptions confirm automatically. Unlike email/HTTPS, they need no manual confirmation step, so a stuck PendingConfirmation means you subscribed the wrong protocol.
Let each subscriber drain independently. A slow or failing consumer on one queue must never block delivery to the others.

B - Ordering & Deduplication (FIFO)

Reach for FIFO only when order or exactly-once-ish delivery matters. It trades throughput for guarantees; standard is faster when order is not a requirement.
Name and configure FIFO resources at creation time. The .fifo suffix and FifoTopic/FifoQueue attribute cannot be added or removed after the fact.
Pick a MessageGroupId granular enough to parallelize. One group for all traffic serializes every consumer; group by customer or order instead.
Prefer ContentBasedDeduplication over hand-rolled dedup ids. Let SQS hash the body rather than tracking your own MessageDeduplicationId values.
Match FIFO topics to FIFO queues only. A FIFO topic rejects standard queues, email, and HTTPS subscribers outright.

# --- Python (boto3) ---
# FIFO naming and attribute are both required, set once at creation.
queue_url = sqs.create_queue(
    QueueName="orders.fifo",
    Attributes={"FifoQueue": "true", "ContentBasedDeduplication": "true"},
)["QueueUrl"]

// --- TypeScript (AWS SDK v3) ---
// FIFO naming and attribute are both required, set once at creation.
const { QueueUrl } = await sqs.send(new CreateQueueCommand({
  QueueName: "orders.fifo",
  Attributes: { FifoQueue: "true", ContentBasedDeduplication: "true" },
}));

C - Failure Handling (Dead-Letter Queues)

Attach a redrive policy to every production queue. No queue should retry a poison message forever with no escape hatch.
Set maxReceiveCount above 1. Allow a handful of retries (commonly 3-5) so transient failures do not permanently exile recoverable messages.
Match FIFO DLQs to FIFO source queues. A FIFO queue cannot target a standard queue as its dead-letter queue.
Alarm on DLQ depth. A CloudWatch alarm on ApproximateNumberOfMessagesVisible catches silent pileups before they go unnoticed for days.
Redrive with StartMessageMoveTask, not manual resends. Bulk-move messages back to the source queue once the underlying bug is fixed.

D - Throughput & Cost (Batching, Long Polling, Visibility)

Batch sends and deletes up to 10 per call. SendMessageBatch/DeleteMessageBatch cut request count (and cost) versus one-at-a-time calls.
Always enable long polling. Set WaitTimeSeconds (up to 20) so receives wait for messages instead of returning empty and billing anyway.
Size visibility timeouts above worst-case processing time. Or extend a specific message's timeout with ChangeMessageVisibility instead of inflating the default for everyone.
Check batch responses for partial failures. A batch call can partially succeed; always inspect Failed alongside Successful.
Make consumers idempotent. Standard queues are at-least-once, so duplicate delivery is expected behavior, not an edge case.

E - Selective Delivery (Filtering)

Filter at the subscription, not by splitting topics. One topic with per-subscriber FilterPolicy rules scales better than a topic per event type.
Always send the attributes your filters check. A filter on eventType matches nothing if publishers never set that attribute.
Match FilterPolicyScope to how the policy is written. MessageAttributes (default) and MessageBody are not interchangeable - set the scope explicitly if you filter on the body.
Treat non-matching deliveries as silent, not erroneous. Publish succeeds regardless of whether any given subscriber's filter matches.
Review filters periodically for drift. An overly broad filter quietly re-creates full unfiltered fan-out load.

FAQs

Where should I start when building a new fan-out system?

When is FIFO worth the throughput trade-off?

What is the single most common messaging mistake?

No dead-letter queue. Without one, a poison message retries indefinitely, burning consumer capacity, or a low maxReceiveCount discards messages that would have succeeded on retry.

How do I avoid processing the same message twice?

Should I always enable long polling?

Yes. There is essentially no downside: it reduces both empty responses and delivery latency compared to short polling.

Do filter policies replace the need for multiple topics?

In most cases yes. A single topic with per-subscription filter policies handles routing subsets of events far more flexibly than maintaining a topic per event type.

Why did my SNS-to-SQS fan-out silently fail?

Do these rules apply equally to FIFO and standard resources?

SNS & SQS Together: Fan-Out Messaging - the wiring habits in group A.
FIFO Topics & Queues - the ordering habits in group B.
Dead-Letter Queues & Redrive Policies - the failure-handling habits in group C.
Batching, Long Polling & Visibility Timeouts - the throughput habits in group D.
Message Filtering with SNS Subscription Filters - the filtering habits in group E.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

SNS & SQS via SDK Best Practices

Search Documentation

SNS & SQS via SDK Best Practices