Observability Best Practices

These are the habits that keep AWS SDK telemetry useful and affordable, instead of noisy and expensive.

Walk the list once now, then treat it as a review checklist before adding logging or tracing to a new service.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run roughly in order of impact: what to log first, then tracing, then sampling and cost control.
Check the boxes against your own code as a quick audit.
Revisit whenever you add a new client, a new service dependency, or notice a telemetry cost spike.

A - What to Log (and What Not To)

Log identifiers and outcomes, not payloads. Capture operation name, request id, duration, and outcome - never the raw request or response body, which can carry secrets or customer data.
Always capture the request id. response["ResponseMetadata"]["RequestId"] (boto3) or response.$metadata.requestId (SDK v3) is the field AWS support needs first.
Use a fixed field set across services. Consistent names (operation, duration_ms, outcome) are what make logs queryable across a fleet, not any single library's features.
Log at the right level. Successes at info, failures at warning/error, so alerting and filtering by severity actually works.

B - Structured Logging

Emit JSON, not free text. json.dumps(...) or a structured logger beats an interpolated sentence for anything you intend to filter or graph later.
Wrap calls in a shared helper. A single logging wrapper used across every client keeps every call site consistent without copy-pasted boilerplate.
Add a correlation id per incoming request. Group every SDK call made while serving one user request under one id, distinct from the AWS request id.
Never log credentials or signed headers. The resolved credentials and the Authorization header must never reach a log line, structured or not.

C - Tracing (X-Ray and OpenTelemetry)

Instrument the client, not the call site. Use captureAWSv3Client / patch(["boto3"]) (X-Ray) or the botocore/AWS SDK OTel instrumentation packages once, so every call is traced automatically.
Pick one tracing approach per service. X-Ray for AWS-only, console-integrated setups; OpenTelemetry when non-AWS dependencies need to share the same trace.
Propagate context across service boundaries. Trace/segment ids need to be forwarded explicitly when a request crosses an HTTP call to another service.
Keep span/subsegment attributes small. An order id or status code is fine; a full item or response body is not - the same rule as structured logs.

D - Sampling Strategy

Never sample at 100% in high-throughput services. Full tracing on every request adds real overhead and cost at scale; sample a representative percentage instead.
Always trace errors, regardless of sampling rate. A sampling rule that keeps all failures and a fraction of successes gives you the highest signal per byte stored.
Sample per-service, not globally. A low-traffic service can afford 100% tracing; a high-traffic one usually cannot - tune sampling rules accordingly rather than using one blanket rate.
Revisit sampling as traffic grows. A rate that was affordable at launch can become a meaningful cost line later; review it periodically, not just once.

E - Cardinality Control

Avoid raw, high-cardinality fields in every log line or span. A raw user id or full object key logged on every single call multiplies storage cost with limited debugging value.
Aggregate before you emit, where possible. Periodic call-count snapshots (per operation) are often more useful than logging every single call at full detail for high-volume operations.
Bucket or truncate free-form values. If you must include a variable string (a key prefix, a status message), consider bucketing it rather than logging the full unique value.
Watch log/trace volume like any other cost. Set budgets or alarms on log ingestion and trace export volume the same way you would on compute spend.

F - Layering the Three Pillars

Start with structured logging. It is the cheapest to add and pays off immediately for single-call debugging.
Add tracing once cross-service timing matters. X-Ray or OpenTelemetry answers "where did the time go" once one log line per service is not enough.
Add an aggregated view once tracing is flowing. CloudWatch Application Signals (or an equivalent APM view) turns individual traces into a service map and SLO metrics - it needs trace data to exist first.
Don't skip straight to the aggregated view. Without logs and traces underneath it, a service map or dashboard has nothing meaningful to show.

FAQs

What is the single most important rule on this list?

Log identifiers and outcomes, never raw payloads. Every other practice here builds on top of that boundary, and getting it wrong is the one mistake with real security and privacy consequences.

Should every service trace at 100%?

No. High-throughput services should sample a percentage of normal traffic while always tracing errors. Low-traffic services can often afford full tracing without meaningful cost.

Why does the request id matter so much?

It is the field AWS support and CloudTrail use to trace a specific call on AWS's side. Without it, an intermittent failure is very hard to correlate to a specific AWS-side event.

What is cardinality, in this context?

The number of distinct values a logged or traced field can take. High-cardinality fields (raw ids, full payload strings) multiply storage and indexing cost far more than a handful of enum-like values.

Do I need X-Ray and OpenTelemetry, or just one?

Just one per service, chosen based on whether your dependencies are AWS-only (X-Ray) or mixed with non-AWS systems (OpenTelemetry). Both produce the trace data an aggregated view needs.

Why instrument the client instead of each call site?

Instrumenting once, at the client, guarantees every call through it gets consistent telemetry automatically. Per-call-site instrumentation is easy to forget and inconsistent across a codebase.

What should trigger a review of sampling rates?

Traffic growth, a noticeable telemetry cost increase, or a new high-volume operation being added. A rate that was fine at launch can quietly become expensive as usage scales.

Is an aggregated view like Application Signals a starting point or an end point?

An end point built on the other layers, not a starting point. It needs logs and trace data flowing first; adding it before instrumenting logging and tracing leaves it with nothing to visualize.

Observability for AWS SDK Applications - the three-pillar model these practices apply to.
Structured Logging Around SDK Calls - the logging rules in groups A and B, in depth.
AWS X-Ray Tracing for SDK-Calling Services - the tracing rules in group C, AWS-native.
OpenTelemetry Instrumentation for AWS SDKs - the tracing rules in group C, vendor-neutral.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Observability Best Practices

These are the habits that keep AWS SDK telemetry useful and affordable, instead of noisy and expensive.

Walk the list once now, then treat it as a review checklist before adding logging or tracing to a new service.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run roughly in order of impact: what to log first, then tracing, then sampling and cost control.
Check the boxes against your own code as a quick audit.
Revisit whenever you add a new client, a new service dependency, or notice a telemetry cost spike.

A - What to Log (and What Not To)

Log identifiers and outcomes, not payloads. Capture operation name, request id, duration, and outcome - never the raw request or response body, which can carry secrets or customer data.
Always capture the request id. response["ResponseMetadata"]["RequestId"] (boto3) or response.$metadata.requestId (SDK v3) is the field AWS support needs first.
Use a fixed field set across services. Consistent names (operation, duration_ms, outcome) are what make logs queryable across a fleet, not any single library's features.
Log at the right level. Successes at info, failures at warning/error, so alerting and filtering by severity actually works.

B - Structured Logging

Emit JSON, not free text. json.dumps(...) or a structured logger beats an interpolated sentence for anything you intend to filter or graph later.
Wrap calls in a shared helper. A single logging wrapper used across every client keeps every call site consistent without copy-pasted boilerplate.
Add a correlation id per incoming request. Group every SDK call made while serving one user request under one id, distinct from the AWS request id.
Never log credentials or signed headers. The resolved credentials and the Authorization header must never reach a log line, structured or not.

C - Tracing (X-Ray and OpenTelemetry)

Instrument the client, not the call site. Use captureAWSv3Client / patch(["boto3"]) (X-Ray) or the botocore/AWS SDK OTel instrumentation packages once, so every call is traced automatically.
Pick one tracing approach per service. X-Ray for AWS-only, console-integrated setups; OpenTelemetry when non-AWS dependencies need to share the same trace.
Propagate context across service boundaries. Trace/segment ids need to be forwarded explicitly when a request crosses an HTTP call to another service.
Keep span/subsegment attributes small. An order id or status code is fine; a full item or response body is not - the same rule as structured logs.

D - Sampling Strategy

Never sample at 100% in high-throughput services. Full tracing on every request adds real overhead and cost at scale; sample a representative percentage instead.
Always trace errors, regardless of sampling rate. A sampling rule that keeps all failures and a fraction of successes gives you the highest signal per byte stored.
Sample per-service, not globally. A low-traffic service can afford 100% tracing; a high-traffic one usually cannot - tune sampling rules accordingly rather than using one blanket rate.
Revisit sampling as traffic grows. A rate that was affordable at launch can become a meaningful cost line later; review it periodically, not just once.

E - Cardinality Control

Avoid raw, high-cardinality fields in every log line or span. A raw user id or full object key logged on every single call multiplies storage cost with limited debugging value.
Aggregate before you emit, where possible. Periodic call-count snapshots (per operation) are often more useful than logging every single call at full detail for high-volume operations.
Bucket or truncate free-form values. If you must include a variable string (a key prefix, a status message), consider bucketing it rather than logging the full unique value.
Watch log/trace volume like any other cost. Set budgets or alarms on log ingestion and trace export volume the same way you would on compute spend.

F - Layering the Three Pillars

Start with structured logging. It is the cheapest to add and pays off immediately for single-call debugging.
Add tracing once cross-service timing matters. X-Ray or OpenTelemetry answers "where did the time go" once one log line per service is not enough.
Add an aggregated view once tracing is flowing. CloudWatch Application Signals (or an equivalent APM view) turns individual traces into a service map and SLO metrics - it needs trace data to exist first.
Don't skip straight to the aggregated view. Without logs and traces underneath it, a service map or dashboard has nothing meaningful to show.

FAQs

What is the single most important rule on this list?

Log identifiers and outcomes, never raw payloads. Every other practice here builds on top of that boundary, and getting it wrong is the one mistake with real security and privacy consequences.

Should every service trace at 100%?

No. High-throughput services should sample a percentage of normal traffic while always tracing errors. Low-traffic services can often afford full tracing without meaningful cost.

Why does the request id matter so much?

It is the field AWS support and CloudTrail use to trace a specific call on AWS's side. Without it, an intermittent failure is very hard to correlate to a specific AWS-side event.

What is cardinality, in this context?

Do I need X-Ray and OpenTelemetry, or just one?

Just one per service, chosen based on whether your dependencies are AWS-only (X-Ray) or mixed with non-AWS systems (OpenTelemetry). Both produce the trace data an aggregated view needs.

Why instrument the client instead of each call site?

Instrumenting once, at the client, guarantees every call through it gets consistent telemetry automatically. Per-call-site instrumentation is easy to forget and inconsistent across a codebase.

What should trigger a review of sampling rates?

Traffic growth, a noticeable telemetry cost increase, or a new high-volume operation being added. A rate that was fine at launch can quietly become expensive as usage scales.

Is an aggregated view like Application Signals a starting point or an end point?

An end point built on the other layers, not a starting point. It needs logs and trace data flowing first; adding it before instrumenting logging and tracing leaves it with nothing to visualize.

Observability for AWS SDK Applications - the three-pillar model these practices apply to.
Structured Logging Around SDK Calls - the logging rules in groups A and B, in depth.
AWS X-Ray Tracing for SDK-Calling Services - the tracing rules in group C, AWS-native.
OpenTelemetry Instrumentation for AWS SDKs - the tracing rules in group C, vendor-neutral.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Observability Best Practices

Search Documentation

Observability Best Practices