Structured Logging Around SDK Calls

A log line that says "S3 call succeeded" is nearly useless at 2am. A log line with the operation name, request id, duration, and outcome as separate fields is something you can filter, graph, and alert on.

Structured logging means emitting logs as key-value data - most commonly JSON - instead of a free-text sentence. Around AWS SDK calls specifically, structured logging turns every call into a small, consistent, queryable record.

This page builds that pattern for both boto3 and AWS SDK v3, and is deliberately narrow about what belongs in the log: identifiers and outcomes, never the request or response bodies themselves.

Recipe

The pattern: one structured log line per SDK call, with a fixed set of fields, emitted from a shared wrapper so every call site is consistent.

# --- Python (boto3) ---
import json
import logging
import time
 
logger = logging.getLogger("app")
 
def log_call(operation: str, duration_ms: float, request_id: str, outcome: str, **extra):
    logger.info(json.dumps({
        "operation": operation,
        "duration_ms": round(duration_ms, 1),
        "request_id": request_id,
        "outcome": outcome,
        **extra,
    }))

// --- TypeScript (AWS SDK v3) ---
function logCall(
  operation: string,
  durationMs: number,
  requestId: string | undefined,
  outcome: "ok" | "error",
  extra: Record<string, unknown> = {},
) {
  console.log(JSON.stringify({
    operation,
    durationMs: Math.round(durationMs * 10) / 10,
    requestId,
    outcome,
    ...extra,
  }));
}

When to reach for this:

• Every service that makes AWS SDK calls and needs its logs to be searchable across many instances.
• Any environment shipping logs to CloudWatch Logs Insights, Datadog, or another log aggregator that benefits from consistent JSON fields.
• Debugging intermittent latency or errors where you need to filter by operation or request id, not read raw text.
• As the foundation layer before adding distributed tracing (X-Ray or OpenTelemetry).

Working Example

Wire the wrapper around a real call, capturing timing and the AWS-assigned request id in both success and failure paths.

# --- Python (boto3) ---
import json
import logging
import time
import boto3
from botocore.exceptions import ClientError
 
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("orders-service")
ddb = boto3.client("dynamodb")
 
def get_order(order_id: str):
    start = time.monotonic()
    try:
        response = ddb.get_item(TableName="Orders", Key={"id": {"S": order_id}})
        duration_ms = (time.monotonic() - start) * 1000
        logger.info(json.dumps({
            "operation": "dynamodb.GetItem",
            "duration_ms": round(duration_ms, 1),
            "request_id": response["ResponseMetadata"]["RequestId"],
            "outcome": "ok",
            "found": "Item" in response,
        }))
        return response.get("Item")
    except ClientError as err:
        duration_ms = (time.monotonic() - start) * 1000
        logger.warning(json.dumps({
            "operation": "dynamodb.GetItem",
            "duration_ms": round(duration_ms, 1),
            "request_id": err.response["ResponseMetadata"]["RequestId"],
            "outcome": "error",
            "error_code": err.response["Error"]["Code"],
        }))
        raise

// --- TypeScript (AWS SDK v3) ---
import { DynamoDBClient, GetItemCommand } from "@aws-sdk/client-dynamodb";
 
const ddb = new DynamoDBClient({});
 
async function getOrder(orderId: string) {
  const start = Date.now();
  try {
    const response = await ddb.send(
      new GetItemCommand({ TableName: "Orders", Key: { id: { S: orderId } } }),
    );
    console.log(JSON.stringify({
      operation: "dynamodb.GetItem",
      durationMs: Date.now() - start,
      requestId: response.$metadata.requestId,
      outcome: "ok",
      found: response.Item !== undefined,
    }));
    return response.Item;
  } catch (err) {
    const e = err as { name: string; $metadata?: { requestId?: string } };
    console.warn(JSON.stringify({
      operation: "dynamodb.GetItem",
      durationMs: Date.now() - start,
      requestId: e.$metadata?.requestId,
      outcome: "error",
      errorCode: e.name,
    }));
    throw err;
  }
}

What this demonstrates:

• The same field set (operation, duration_ms, request_id, outcome) appears whether the call succeeds or fails.
• found and similar boolean/enum fields describe the shape of the result without logging the item itself.
• The error path logs the AWS error code and still emits the request id - the single most useful field for support cases.
• Neither branch logs response["Item"] or response.Item, since order records may carry customer data.

Deep Dive

Picking a JSON Logger

• Python's standard logging module can emit JSON with json.dumps(...) as shown, or via a formatter library like python-json-logger for less boilerplate at scale.
• TypeScript has no bundled structured logger; pino and winston are common choices, or JSON.stringify directly for a minimal footprint as shown here.
• Whichever you choose, keep the field names fixed across your codebase - a log aggregator's value comes from consistency, not from any single library's features.
• Set the logger's base level (info for success, warning/error for failures) so filtering by severity works without parsing the JSON body.

A Consistent Field Set

Adding your own correlation id (from an incoming request) alongside these fields lets you group every SDK call made while serving one user request.

What Never Belongs in the Log

• Request bodies: item attributes, object contents, message payloads. These can carry customer data or secrets.
• Response bodies beyond a small derived field (like found above). Log a summary, not the record.
• Credentials, tokens, or signed headers - the Authorization header and any resolved credentials should never reach a log line. See the security-best-practices section's guidance on secrets handling for the broader pattern.
• Full stack traces at info level; reserve those for error level and consider redacting inner exception args that may echo input parameters.

Wiring It Into a Shared Wrapper

Both examples above repeat the same shape at each call site. In practice, wrap this once (as shown in Observability Basics) and reuse it for every client, so a new call site gets structured logging for free rather than by convention alone.

Gotchas

• Logging the whole response object - logger.info(response) or console.log(response) can dump the entire item or object body. Fix: log named, derived fields only.
• Inconsistent field names across services - one team logs latency_ms, another duration_ms. Fix: agree on one field set and enforce it in a shared logging helper.
• Free-text log messages with embedded values - f"call took {duration}ms" is not filterable. Fix: always emit structured key-value pairs, JSON at minimum.
• Missing request id on errors - some error paths only log the exception message. Fix: always extract request_id / requestId from the error response, not just from the success path.
• Logging at the wrong level - logging every successful call at error level (or every real error at debug) breaks alerting. Fix: map outcome to log level deliberately.
• High-cardinality fields in every log line - logging a raw user id or full object key on every call can blow up log storage costs at scale. Fix: log identifiers you actually query on, and consider sampling high-volume, low-value log lines.

Alternatives

FAQs

Related

• Observability Basics - the request-id-and-latency pattern this page formalizes.
• AWS X-Ray Tracing for SDK-Calling Services - adding distributed tracing on top of these logs.
• OpenTelemetry Instrumentation for AWS SDKs - the vendor-neutral tracing alternative.
• Secrets Handling: Never in Code, Never in Logs - the fuller rule behind "never log the payload."

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).