CloudWatch Logs Insights Queries via SDK

CloudWatch Logs Insights lets you search and aggregate log data with a purpose-built query language, without a metric filter defined in advance. Because a query can scan large volumes of logs, the SDK models it as an asynchronous job: you start it, then poll for results.

This page covers that async pattern end to end.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3, time
 
logs = boto3.client("logs", region_name="us-east-1")
 
start = logs.start_query(
    logGroupName="/myapp/checkout",
    startTime=int(time.time()) - 3600,
    endTime=int(time.time()),
    queryString="fields @timestamp, @message | filter @message like /ERROR/ | limit 20",
)
query_id = start["queryId"]

// --- TypeScript (AWS SDK v3) ---
import { CloudWatchLogsClient, StartQueryCommand } from "@aws-sdk/client-cloudwatch-logs";
 
const logs = new CloudWatchLogsClient({ region: "us-east-1" });
const now = Math.floor(Date.now() / 1000);
 
const { queryId } = await logs.send(new StartQueryCommand({
  logGroupName: "/myapp/checkout",
  startTime: now - 3600,
  endTime: now,
  queryString: "fields @timestamp, @message | filter @message like /ERROR/ | limit 20",
}));

When to reach for this:

• Ad hoc investigation of log content you did not pre-declare as a metric filter.
• Debugging a production incident by searching across many log streams at once.
• Building a report that aggregates log fields with stats (counts, averages, percentiles).
• Programmatically extracting structured data buried in unstructured log text.

Working Example

Start a query, poll until it completes, then read the results.

# --- Python (boto3) ---
import boto3, time
 
logs = boto3.client("logs", region_name="us-east-1")
 
# 1. Start the query.
start = logs.start_query(
    logGroupName="/myapp/checkout",
    startTime=int(time.time()) - 3600,
    endTime=int(time.time()),
    queryString="""
        fields @timestamp, @message
        | filter @message like /ERROR/
        | stats count(*) as errorCount by bin(5m)
    """,
)
query_id = start["queryId"]
 
# 2. Poll until the query finishes.
while True:
    result = logs.get_query_results(queryId=query_id)
    if result["status"] in ("Complete", "Failed", "Cancelled", "Timeout"):
        break
    time.sleep(1)
 
# 3. Read the results.
print(result["status"])
for row in result["results"]:
    fields = {f["field"]: f["value"] for f in row}
    print(fields)

// --- TypeScript (AWS SDK v3) ---
import {
  CloudWatchLogsClient, StartQueryCommand, GetQueryResultsCommand,
} from "@aws-sdk/client-cloudwatch-logs";
 
const logs = new CloudWatchLogsClient({ region: "us-east-1" });
const now = Math.floor(Date.now() / 1000);
 
// 1. Start the query.
const { queryId } = await logs.send(new StartQueryCommand({
  logGroupName: "/myapp/checkout",
  startTime: now - 3600,
  endTime: now,
  queryString: `
    fields @timestamp, @message
    | filter @message like /ERROR/
    | stats count(*) as errorCount by bin(5m)
  `,
}));
 
// 2. Poll until the query finishes.
let result;
do {
  result = await logs.send(new GetQueryResultsCommand({ queryId }));
  if (["Complete", "Failed", "Cancelled", "Timeout"].includes(result.status ?? "")) break;
  await new Promise((r) => setTimeout(r, 1000));
} while (true);
 
// 3. Read the results.
console.log(result.status);
for (const row of result.results ?? []) {
  const fields = Object.fromEntries((row ?? []).map((f) => [f.field, f.value]));
  console.log(fields);
}

What this demonstrates:

• StartQuery returns immediately with a queryId - the query keeps running server-side.
• GetQueryResults is called repeatedly; its status field tells you when to stop polling.
• The query string (fields, filter, stats ... by bin(...)) is the same plain string in both SDKs, since Logs Insights syntax is not SDK-specific.
• Each result row is a list of {field, value} pairs, not a flat object - both languages reshape it for convenience.

Deep Dive

The Async Query Lifecycle

A Logs Insights query has five possible terminal-adjacent states: Scheduled, Running, Complete, Failed, Cancelled, and Timeout. Only Complete means you have final results; the others mean stop polling and handle the outcome.

StartQuery accepts one log group name (or, in newer API versions, multiple logGroupNames/logGroupIdentifiers) plus a Unix startTime/endTime range and the query string. It does not accept the query language as a structured object - it is always a plain string.

The Query Language Itself

Logs Insights queries are piped commands, similar in spirit to a Unix shell pipeline:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc
| limit 20

• fields selects or computes columns, including the built-in @timestamp, @message, @logStream fields.
• filter narrows rows, similar to a WHERE clause.
• stats aggregates, commonly with count(), avg(), pct(), grouped by a field or a time bucket like bin(5m).
• sort and limit shape the final output.

This syntax is identical whether you submit it from boto3, SDK v3, the CLI, or the Console query editor - it is not part of either language's SDK surface, just a string you pass through.

Stopping a Long-Running Query

If a query is scanning more data than expected, cancel it rather than waiting out its scan limit.

# --- Python (boto3) ---
logs.stop_query(queryId=query_id)

// --- TypeScript (AWS SDK v3) ---
import { StopQueryCommand } from "@aws-sdk/client-cloudwatch-logs";
 
await logs.send(new StopQueryCommand({ queryId }));

StopQuery transitions the query to Cancelled; a subsequent GetQueryResults call still returns whatever partial results were computed before cancellation.

Querying Across Multiple Log Groups

A single query can span multiple log groups at once, useful when a request fans out across several Lambda functions or services.

# --- Python (boto3) ---
logs.start_query(
    logGroupNames=["/myapp/checkout", "/myapp/payments"],
    startTime=int(time.time()) - 3600,
    endTime=int(time.time()),
    queryString="fields @timestamp, @message, @log | filter @message like /ERROR/",
)

// --- TypeScript (AWS SDK v3) ---
await logs.send(new StartQueryCommand({
  logGroupNames: ["/myapp/checkout", "/myapp/payments"],
  startTime: now - 3600,
  endTime: now,
  queryString: "fields @timestamp, @message, @log | filter @message like /ERROR/",
}));

The @log field in results tells you which log group each row came from.

Gotchas

• Treating GetQueryResults as blocking - it returns immediately with whatever progress exists so far. Fix: poll in a loop and check status until it reaches a terminal value.
• Fixed sleep loops with no upper bound - a stuck query polls forever. Fix: cap total polling attempts or elapsed time, and call StopQuery if exceeded.
• Passing the query language as a structured parameter - there is no field-by-field query builder. Fix: build the pipeline string yourself; treat it as opaque text to the SDK.
• Overly broad time ranges - a wide startTime/endTime window over a busy log group can hit the per-query data scan limit before returning useful results. Fix: narrow the range or add an early filter.
• Forgetting limit - without it, stats and fields queries can return far more rows than needed. Fix: always cap output with limit unless you specifically want everything.
• Ignoring Failed/Timeout status - silently treating any non-Complete status as "still running." Fix: branch explicitly on Failed, Cancelled, and Timeout and handle each.

Alternatives

FAQs

Related

• CloudWatch's Metrics/Logs/Alarms/Events Model - where the log pillar fits overall.
• Custom Metrics & Embedded Metric Format (EMF) - the log lines Logs Insights can search.
• CloudWatch Deep Dive Basics - the quick-tour version of a Logs Insights query.
• CloudWatch via SDK Best Practices - query-cost and time-range guidance.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).