Asynchronous Inference for Large Payloads

A synchronous InvokeEndpoint call holds an open connection until the container answers. That works fine for a small request answered in milliseconds - it works badly for a multi-megabyte document or a prediction that takes tens of seconds, where a caller holding a connection open that long is fragile and wasteful.

Asynchronous inference decouples the request from the response. The caller submits input already sitting in S3, gets an output location back immediately, and checks that location - or waits for a notification - once the prediction is actually ready. The endpoint underneath is still instance-backed; only the invocation contract changes.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
sm = boto3.client("sagemaker", region_name="us-east-1")
 
sm.create_endpoint_config(
    EndpointConfigName="doc-classifier-async-config",
    ProductionVariants=[{
        "VariantName": "AllTraffic",
        "ModelName": "doc-classifier-model",
        "InstanceType": "ml.m5.xlarge",
        "InitialInstanceCount": 1,
    }],
    AsyncInferenceConfig={
        "OutputConfig": {
            "S3OutputPath": "s3://my-ml-bucket/async-output/",
            "NotificationConfig": {
                "SuccessTopic": "arn:aws:sns:us-east-1:111122223333:async-inference-success",
                "ErrorTopic": "arn:aws:sns:us-east-1:111122223333:async-inference-error",
            },
        },
        "ClientConfig": {"MaxConcurrentInvocationsPerInstance": 4},
    },
)
 
sm.create_endpoint(
    EndpointName="doc-classifier-async-endpoint",
    EndpointConfigName="doc-classifier-async-config",
)

// --- TypeScript (AWS SDK v3) ---
import {
  SageMakerClient,
  CreateEndpointConfigCommand,
  CreateEndpointCommand,
} from "@aws-sdk/client-sagemaker";
 
const sm = new SageMakerClient({ region: "us-east-1" });
 
await sm.send(new CreateEndpointConfigCommand({
  EndpointConfigName: "doc-classifier-async-config",
  ProductionVariants: [{
    VariantName: "AllTraffic",
    ModelName: "doc-classifier-model",
    InstanceType: "ml.m5.xlarge",
    InitialInstanceCount: 1,
  }],
  AsyncInferenceConfig: {
    OutputConfig: {
      S3OutputPath: "s3://my-ml-bucket/async-output/",
      NotificationConfig: {
        SuccessTopic: "arn:aws:sns:us-east-1:111122223333:async-inference-success",
        ErrorTopic: "arn:aws:sns:us-east-1:111122223333:async-inference-error",
      },
    },
    ClientConfig: { MaxConcurrentInvocationsPerInstance: 4 },
  },
}));
 
await sm.send(new CreateEndpointCommand({
  EndpointName: "doc-classifier-async-endpoint",
  EndpointConfigName: "doc-classifier-async-config",
}));

When to reach for this:

• Payloads too large for a synchronous request body, or inference that routinely takes many seconds or longer.
• Workloads where the caller can act on a result later rather than needing it inline in the same request.
• Avoiding client-side timeout tuning that a very slow synchronous InvokeEndpoint call would otherwise require.

Working Example

Submit an async inference request with input already in S3, then poll the output location for the result.

# --- Python (boto3) ---
import time
 
runtime = boto3.client("sagemaker-runtime", region_name="us-east-1")
 
response = runtime.invoke_endpoint_async(
    EndpointName="doc-classifier-async-endpoint",
    InputLocation="s3://my-ml-bucket/async-input/doc-0042.json",
    ContentType="application/json",
)
output_location = response["OutputLocation"]
print("submitted, result will land at:", output_location)
 
s3 = boto3.client("s3")
bucket, key = output_location.replace("s3://", "").split("/", 1)
for _ in range(20):
    try:
        obj = s3.get_object(Bucket=bucket, Key=key)
        print("result:", obj["Body"].read().decode("utf-8"))
        break
    except s3.exceptions.NoSuchKey:
        time.sleep(15)  # not ready yet - poll again

// --- TypeScript (AWS SDK v3) ---
import { SageMakerRuntimeClient, InvokeEndpointAsyncCommand } from "@aws-sdk/client-sagemaker-runtime";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
 
const runtime = new SageMakerRuntimeClient({ region: "us-east-1" });
const s3 = new S3Client({ region: "us-east-1" });
 
const response = await runtime.send(new InvokeEndpointAsyncCommand({
  EndpointName: "doc-classifier-async-endpoint",
  InputLocation: "s3://my-ml-bucket/async-input/doc-0042.json",
  ContentType: "application/json",
}));
const outputLocation = response.OutputLocation!;
console.log("submitted, result will land at:", outputLocation);
 
const [bucket, ...keyParts] = outputLocation.replace("s3://", "").split("/");
const key = keyParts.join("/");
 
for (let i = 0; i < 20; i++) {
  try {
    const obj = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
    console.log("result:", await obj.Body!.transformToString());
    break;
  } catch {
    await new Promise((r) => setTimeout(r, 15000)); // not ready yet - poll again
  }
}

What this demonstrates:

• InvokeEndpointAsync returns almost immediately with OutputLocation - it never blocks waiting for the container to finish.
• The result only appears in S3 once the container has actually written it; polling before then correctly finds nothing yet.
• With NotificationConfig set, the same result could instead be picked up from an SNS subscription rather than polled - polling here is just the simplest illustration.
• Nothing about the submission call tells you how long the prediction will take - that depends entirely on the container and the input.

Deep Dive

It's a queue in front of a normal endpoint

Under AsyncInferenceConfig, the endpoint still runs on instance-backed ProductionVariants exactly like a real-time endpoint - the same InstanceType/InitialInstanceCount fields apply, and the same auto scaling from Real-Time Endpoints & Auto Scaling works underneath it. What AsyncInferenceConfig adds is a managed queue in front: requests submitted via InvokeEndpointAsync wait in that queue until an instance is free, rather than being processed synchronously against an open connection.

Input and output both live in S3

Unlike a synchronous call, the request body isn't sent inline - InputLocation points at an object already in S3, which is what makes large payloads practical: there's no HTTP request-body size constraint to work around. The response follows the same pattern in reverse: the container's output is written to OutputConfig.S3OutputPath, under a key derived from the request, rather than returned directly to the caller.

Two ways to know a result is ready

NotificationConfig's SuccessTopic and ErrorTopic push an SNS message the moment a result (or a failure) is written, which is the more efficient path for most integrations - subscribe once, react to messages as they arrive. Polling S3 directly, as in the example above, works too and needs no extra AWS resources, but costs more requests and adds latency between when a result lands and when a poller notices it.

Concurrency is bounded per instance, not per endpoint

ClientConfig.MaxConcurrentInvocationsPerInstance caps how many requests a single instance processes at once - it's a per-instance setting, so the endpoint's total in-flight capacity is this value multiplied by however many instances auto scaling currently has running. Setting it too high risks resource contention on the instance; too low leaves capacity underused while requests queue.

Gotchas

• Sending an inline request body. Async inference expects InputLocation (an S3 URI), not a Body payload - the whole design assumes the input already lives in S3. Fix: upload the request payload to S3 first, then reference it by InputLocation.
• Expecting an immediate result. InvokeEndpointAsync's response is an acknowledgment, not a prediction - the actual output isn't written until the queued request is processed. Fix: poll OutputLocation or subscribe to the SNS topics; never assume the result exists right after submission.
• Skipping NotificationConfig and polling too aggressively. Tight polling loops against S3 waste requests and add cost without speeding up the actual inference. Fix: prefer SNS notifications, or poll on a sensible interval with backoff.
• Forgetting the endpoint still bills per instance-hour. Async inference changes how you invoke the endpoint, not its underlying compute cost - idle instances behind an async endpoint still bill continuously. Fix: pair async inference with auto scaling (including scale-to-zero-adjacent low minimums where acceptable) if traffic is intermittent.
• Not handling the error path. A failed inference writes to a different location (or fires ErrorTopic) than a successful one - code that only checks the success path silently misses failures. Fix: check both SuccessTopic/ErrorTopic or inspect the failure output object explicitly.

Alternatives

FAQs

Related

• SageMaker's Inference Deployment Options - where async inference fits among the other endpoint types.
• Real-Time Endpoints & Auto Scaling - the instance-backed foundation async inference builds on.
• Serverless Inference for Bursty Workloads - a different answer to avoiding idle instance cost.
• SageMaker Deployment via SDK Best Practices - checklist covering endpoint-type choice.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).