Search across all documentation pages
A trained model does nothing until something serves it. SageMaker's answer is not one "deploy" button - it is four distinct endpoint shapes plus a fifth option that isn't an endpoint at all, each suited to a different traffic pattern.
This page maps the terrain before the rest of this section goes deep on each option. Once you can place real-time, serverless, asynchronous, and multi-model/multi-container endpoints against the traffic pattern they were built for, the individual CreateEndpointConfig fields in later pages read as deliberate trade-offs instead of an arbitrary pile of parameters.
CreateModel / CreateEndpointConfig / CreateEndpoint sequence - the choice is expressed entirely through which fields you set on a ProductionVariant.Every SageMaker inference option starts the same way: CreateModel registers a model artifact (from a training job) plus a serving container image. From there, CreateEndpointConfig describes how that model should be hosted, and CreateEndpoint stands the configuration up as a running resource. What changes between the four live-endpoint options is entirely inside ProductionVariants - the same field that names the model and, in a real-time config, its instance type and count.
Real-time endpoints are the default mental model: one or more dedicated instances sit behind an HTTPS endpoint, ready to answer InvokeEndpoint calls in milliseconds. ProductionVariants[].InstanceType and InitialInstanceCount pick the compute. The trade-off is billing - instances run, and bill, continuously whether or not a request ever arrives.
Serverless inference replaces instance sizing with ServerlessConfig{MemorySizeInMB, MaxConcurrency} on the variant. There is no instance to provision or pay for between requests; AWS manages capacity behind the scenes. The trade-off is a cold start on the first request after idle time, and less control over exact latency.
Asynchronous inference adds AsyncInferenceConfig to the endpoint config alongside an instance-based variant. Instead of returning a prediction synchronously, InvokeEndpointAsync accepts a pointer to the input in S3, queues the request, and returns immediately with a location to poll for the result. It exists for payloads or processing times that don't fit a synchronous HTTP call.
Multi-model and multi-container endpoints relax the assumption that one endpoint serves one model. A multi-model endpoint (Mode: "MultiModel" on the container, with MultiModelConfig) dynamically loads models from a shared S3 prefix and routes each InvokeEndpoint call via a TargetModel header. A multi-container endpoint instead hosts several distinct containers (Containers[] on CreateModel, each with a ContainerHostname) and routes via TargetContainerHostname or a fixed serial pipeline.
Batch transform (CreateTransformJob) is the odd one out - it isn't a live endpoint at all. It reads a static dataset from S3, runs inference over it once, writes results back to S3, and then the job's compute goes away. There is no InvokeEndpoint call and nothing to keep billing afterward.
The shared spine - CreateModel, CreateEndpointConfig, CreateEndpoint - means switching between real-time, serverless, and async inference is mostly a matter of swapping which fields appear on the ProductionVariant, not learning a new API family. A model registered once with CreateModel can, in principle, sit behind any of these endpoint types by changing only the endpoint config that references it.
Multi-model and multi-container endpoints interact with this spine slightly differently: the branching happens on the container side (Mode: "MultiModel", or a Containers list) rather than purely on the ProductionVariant, because the endpoint is now routing among multiple model artifacts or container images instead of hosting a single one.
Async inference is the one option that changes the invocation contract itself. Real-time, serverless, and multi-model/multi-container endpoints are all invoked through the same synchronous InvokeEndpoint call in the sagemaker-runtime client (with TargetModel or TargetContainerHostname added as needed). Async inference instead uses InvokeEndpointAsync, which returns immediately with an OutputLocation - the caller polls S3 (or listens for an SNS notification) rather than waiting on an open HTTP connection.
Auto scaling sits alongside real-time and async endpoints as a separate, opt-in layer through Application Auto Scaling, registering the endpoint's variant as a scalable target. Serverless inference and multi-model endpoints don't need this layer - their scaling behavior (concurrency limits, dynamic model loading) is handled inside the endpoint config itself.
In production, the choice usually comes down to three questions in order: is traffic frequent and latency-sensitive enough to justify paying for idle capacity (real-time); is traffic bursty or unpredictable enough that idle capacity would mostly go unused (serverless); and is the workload not latency-sensitive at all - large payloads, long processing times, or both (async)? Multi-model and multi-container endpoints answer a different question entirely: not "how should one model be served" but "how do I serve many models or many containers without paying for a dedicated endpoint per model."
| Option | Billing shape | Typical latency | Best fit |
|---|---|---|---|
| Real-time | Continuous, per instance-hour | Milliseconds | Steady, latency-sensitive traffic |
| Serverless | Per-inference, no idle cost | Seconds (cold start possible) | Bursty or intermittent traffic |
| Asynchronous | Continuous instance behind a queue | Seconds to minutes | Large payloads or slow inference |
| Multi-model/multi-container | Continuous, shared across models | Milliseconds (cache-dependent) | Many models, uneven per-model traffic |
| Batch transform | Job-duration only | N/A - not live | One-off or scheduled offline scoring |
A model already trained and packaged for one option is not locked into it - the model artifact and container are the constant; only the endpoint config around them differs, which is why teams frequently start with a real-time endpoint for development and move to serverless or multi-model in production once the actual traffic shape is known.
CreateModel doesn't depend on which endpoint config will reference it.A real-time endpoint, for development and steady low-to-moderate traffic. It's the simplest mental model and the one every other option's CreateEndpointConfig shape is compared against.
ServerlessConfig{MemorySizeInMB, MaxConcurrency} on the ProductionVariant, in place of InstanceType/InitialInstanceCount. Everything else about CreateEndpointConfig stays the same shape.
Multi-model and multi-container endpoints do - via InvokeEndpoint with a TargetModel or TargetContainerHostname header. Async inference uses a different call, InvokeEndpointAsync, because it doesn't return a synchronous response.
Yes, whenever there's no live traffic to serve - a nightly scoring run over a fixed dataset, for example. It avoids paying for any endpoint at all, since the job's compute exists only for its duration.
Yes. The model artifact registered with CreateModel isn't tied to an endpoint type - you create a new EndpointConfig with the shape you want and point a new or updated endpoint at the same model.
Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).
Reviewed by Chris St. John·Last updated Jul 24, 2026