Real-Time Endpoints & Auto Scaling

A real-time endpoint answers InvokeEndpoint calls in milliseconds because it keeps instances running continuously. That is also its cost story - those instances bill hourly whether traffic is heavy, light, or absent.

Auto scaling is how you reconcile the two: keep enough instances up to meet latency under load, without paying for peak capacity around the clock. SageMaker doesn't scale endpoints itself - it hands that job to Application Auto Scaling, a separate AWS service that watches a metric and adjusts InstanceCount on your behalf.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
sm = boto3.client("sagemaker", region_name="us-east-1")
aas = boto3.client("application-autoscaling", region_name="us-east-1")
 
resource_id = "endpoint/xgboost-basics-endpoint/variant/AllTraffic"
 
# Register the endpoint variant as a scalable target.
aas.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=1,
    MaxCapacity=4,
)
 
# Attach a target-tracking policy on invocations per instance.
aas.put_scaling_policy(
    PolicyName="xgboost-basics-target-tracking",
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 70.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance",
        },
        "ScaleInCooldown": 300,
        "ScaleOutCooldown": 60,
    },
)

// --- TypeScript (AWS SDK v3) ---
import {
  ApplicationAutoScalingClient,
  RegisterScalableTargetCommand,
  PutScalingPolicyCommand,
} from "@aws-sdk/client-application-auto-scaling";
 
const aas = new ApplicationAutoScalingClient({ region: "us-east-1" });
const resourceId = "endpoint/xgboost-basics-endpoint/variant/AllTraffic";
 
// Register the endpoint variant as a scalable target.
await aas.send(new RegisterScalableTargetCommand({
  ServiceNamespace: "sagemaker",
  ResourceId: resourceId,
  ScalableDimension: "sagemaker:variant:DesiredInstanceCount",
  MinCapacity: 1,
  MaxCapacity: 4,
}));
 
// Attach a target-tracking policy on invocations per instance.
await aas.send(new PutScalingPolicyCommand({
  PolicyName: "xgboost-basics-target-tracking",
  ServiceNamespace: "sagemaker",
  ResourceId: resourceId,
  ScalableDimension: "sagemaker:variant:DesiredInstanceCount",
  PolicyType: "TargetTrackingScaling",
  TargetTrackingScalingPolicyConfiguration: {
    TargetValue: 70.0,
    PredefinedMetricSpecification: {
      PredefinedMetricType: "SageMakerVariantInvocationsPerInstance",
    },
    ScaleInCooldown: 300,
    ScaleOutCooldown: 60,
  },
}));

When to reach for this:

• A real-time endpoint whose traffic varies enough that a fixed instance count either wastes money at quiet times or misses latency targets at peak.
• Standardizing a scaling policy across endpoints instead of hand-tuning instance counts per deployment.
• Reducing on-call toil from manually resizing endpoints during traffic events.

Working Example

Confirm the scalable target and policy are attached, then read back the endpoint's current instance count.

# --- Python (boto3) ---
targets = aas.describe_scalable_targets(
    ServiceNamespace="sagemaker",
    ResourceIds=[resource_id],
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
)
print("registered:", targets["ScalableTargets"][0]["MinCapacity"], "-", targets["ScalableTargets"][0]["MaxCapacity"])
 
desc = sm.describe_endpoint(EndpointName="xgboost-basics-endpoint")
for variant in desc["ProductionVariants"]:
    print(variant["VariantName"], "->", variant["CurrentInstanceCount"], "instances")

// --- TypeScript (AWS SDK v3) ---
import { DescribeScalableTargetsCommand } from "@aws-sdk/client-application-auto-scaling";
import { DescribeEndpointCommand } from "@aws-sdk/client-sagemaker";
 
const targets = await aas.send(new DescribeScalableTargetsCommand({
  ServiceNamespace: "sagemaker",
  ResourceIds: [resourceId],
  ScalableDimension: "sagemaker:variant:DesiredInstanceCount",
}));
console.log("registered:", targets.ScalableTargets?.[0].MinCapacity, "-", targets.ScalableTargets?.[0].MaxCapacity);
 
const desc = await sm.send(new DescribeEndpointCommand({ EndpointName: "xgboost-basics-endpoint" }));
for (const variant of desc.ProductionVariants ?? []) {
  console.log(variant.VariantName, "->", variant.CurrentInstanceCount, "instances");
}

What this demonstrates:

• The scalable target and the endpoint are two different resources tracked by two different services - checking both confirms the wiring is correct.
• CurrentInstanceCount on DescribeEndpoint reflects what auto scaling has actually provisioned, which can differ from InitialInstanceCount once scaling has acted.
• Nothing here tells you whether the target value (70 invocations/instance) is well-tuned - that comes from watching real traffic and CloudWatch metrics over time.
• A scaling action itself takes a few minutes to add or remove instances - it is not instantaneous.

Deep Dive

Why auto scaling is a separate service

SageMaker endpoints don't scale themselves - Application Auto Scaling is a shared AWS service that also scales DynamoDB tables, ECS services, and other resources, using the same RegisterScalableTarget/PutScalingPolicy calls with a different ServiceNamespace and ScalableDimension. For SageMaker, the resource is identified as endpoint/<endpoint-name>/variant/<variant-name>, and the dimension is sagemaker:variant:DesiredInstanceCount. This is why scaling logic lives in a different client than the one used to create the endpoint.

Target tracking vs step scaling

TargetTrackingScaling is the simplest policy type: pick a metric and a target value, and Application Auto Scaling adds or removes instances to keep the metric near that target. SageMakerVariantInvocationsPerInstance is the predefined metric built for this - it normalizes traffic per instance, so the target value represents "how many requests per instance per minute is comfortable," not a raw endpoint-wide number.

A step scaling policy exists as an alternative, defining explicit scaling adjustments per CloudWatch alarm threshold. It gives finer control but requires you to manage alarms and step adjustments directly rather than a single target value - most endpoints don't need that complexity.

Cooldowns and asymmetric reaction

ScaleOutCooldown and ScaleInCooldown control how long Application Auto Scaling waits after a scaling action before considering another one. Scaling out (adding instances) typically uses a short cooldown to react quickly to a traffic spike and protect latency; scaling in (removing instances) typically uses a longer one, since removing capacity too eagerly risks a scale-out/scale-in flap under bursty-but-not-sustained traffic.

Min/max capacity as a cost ceiling and latency floor

MinCapacity guarantees a floor - the endpoint will never scale below it, which matters if a cold instance start would violate a latency requirement on the next request. MaxCapacity is a cost ceiling: however aggressive the metric or however heavy real traffic gets, the endpoint will not provision more instances than this, which caps worst-case billing at the price of a possible latency degradation under extreme load.

Gotchas

• Registering the scalable target against the wrong ResourceId format. It must be exactly endpoint/<endpoint-name>/variant/<variant-name>, matching the VariantName set in CreateEndpointConfig. Fix: copy the variant name directly from the endpoint config rather than assuming "AllTraffic" everywhere.
• Setting MinCapacity to 0. A real-time endpoint's variant cannot scale to zero instances - MinCapacity must be at least 1. Fix: use serverless inference if you need true scale-to-zero behavior.
• Expecting instant scaling. A target-tracking policy reacts to sustained metric pressure, not an instantaneous spike - it can take minutes to provision new instances. Fix: size MinCapacity to absorb a typical burst before scaling has time to react.
• Tuning TargetValue without real traffic data. A target picked before seeing production load is a guess. Fix: start conservative, then adjust the target value based on observed SageMakerVariantInvocationsPerInstance and latency together.
• Forgetting the policy is per-variant. A multi-variant endpoint (e.g. for canary testing) needs a separate scalable target and policy per variant. Fix: register and tune each VariantName independently.

Alternatives

FAQs

Related

• SageMaker Deployment Basics - creating the real-time endpoint this page scales.
• SageMaker's Inference Deployment Options - where real-time fits among the other endpoint types.
• Serverless Inference for Bursty Workloads - the scale-to-zero alternative.
• SageMaker Deployment via SDK Best Practices - sizing and scaling checklist.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).