Provisioned Throughput & On-Demand Inference

Capacity models for predictable-latency vs bursty workloads.

Bedrock offers two ways to pay for and reserve inference capacity. On-demand bills per token on shared infrastructure with no commitment - the default nearly everyone should start with. Provisioned Throughput reserves dedicated capacity ("model units") for a model, giving steady, predictable performance at high volume, billed by the hour. The inference call is the same either way; what changes is the modelId you pass and how you are billed.

Cost warning: Provisioned Throughput is a real financial commitment. Reserved model units bill by the hour (often with a required term) whether or not you send any requests. Do not create it to experiment - use on-demand for development and delete provisioned units you are not using.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
# On-demand: just call the model id. No reservation, pay per token.
brt = boto3.client("bedrock-runtime", region_name="us-east-1")
resp = brt.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hi"}]}],
)
print(resp["output"]["message"]["content"][0]["text"])

// --- TypeScript (AWS SDK v3) ---
import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";
 
// On-demand: just call the model id. No reservation, pay per token.
const brt = new BedrockRuntimeClient({ region: "us-east-1" });
const resp = await brt.send(new ConverseCommand({
  modelId: "amazon.nova-lite-v1:0",
  messages: [{ role: "user", content: [{ text: "Hi" }] }],
}));
console.log(resp.output?.message?.content?.[0]?.text);

When to reach for each:

• On-demand - development, spiky or unpredictable traffic, low-to-moderate volume, and getting started. This is the default.
• Provisioned Throughput - steady high-volume production traffic that needs predictable latency and guaranteed capacity.
• Provisioned may also be required to serve certain customized (fine-tuned) models.
• Start on-demand; move to provisioned only when volume and latency requirements justify the committed cost.

Working Example

Provisioning is a control-plane action on the bedrock client (not bedrock-runtime). You create a provisioned model, wait until it is InService, then invoke it by passing its ARN as the modelId.

# --- Python (boto3) ---
import boto3
 
bedrock = boto3.client("bedrock", region_name="us-east-1")          # control plane
brt = boto3.client("bedrock-runtime", region_name="us-east-1")      # data plane
 
# 1. Reserve capacity (BILLS BY THE HOUR from creation). modelUnits = throughput units.
created = bedrock.create_provisioned_model_throughput(
    provisionedModelName="orders-assistant-pt",
    modelId="amazon.nova-lite-v1:0",   # base model id to reserve capacity for
    modelUnits=1,
)
arn = created["provisionedModelArn"]
 
# 2. Wait until it is ready (poll status until "InService").
status = bedrock.get_provisioned_model_throughput(provisionedModelId=arn)["status"]
print("status:", status)
 
# 3. Invoke it: pass the ARN as modelId. The inference call is otherwise identical.
resp = brt.converse(
    modelId=arn,
    messages=[{"role": "user", "content": [{"text": "Ping"}]}],
)
print(resp["output"]["message"]["content"][0]["text"])
 
# 4. Clean up when done so it stops billing.
bedrock.delete_provisioned_model_throughput(provisionedModelId=arn)

// --- TypeScript (AWS SDK v3) ---
import {
  BedrockClient, CreateProvisionedModelThroughputCommand,
  GetProvisionedModelThroughputCommand, DeleteProvisionedModelThroughputCommand,
} from "@aws-sdk/client-bedrock";
import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";
 
const bedrock = new BedrockClient({ region: "us-east-1" });         // control plane
const brt = new BedrockRuntimeClient({ region: "us-east-1" });      // data plane
 
// 1. Reserve capacity (BILLS BY THE HOUR from creation).
const created = await bedrock.send(new CreateProvisionedModelThroughputCommand({
  provisionedModelName: "orders-assistant-pt",
  modelId: "amazon.nova-lite-v1:0",  // base model id to reserve capacity for
  modelUnits: 1,
}));
const arn = created.provisionedModelArn!;
 
// 2. Wait until it is ready (poll status until "InService").
const desc = await bedrock.send(new GetProvisionedModelThroughputCommand({ provisionedModelId: arn }));
console.log("status:", desc.status);
 
// 3. Invoke it: pass the ARN as modelId. The inference call is otherwise identical.
const resp = await brt.send(new ConverseCommand({
  modelId: arn,
  messages: [{ role: "user", content: [{ text: "Ping" }] }],
}));
console.log(resp.output?.message?.content?.[0]?.text);
 
// 4. Clean up when done so it stops billing.
await bedrock.send(new DeleteProvisionedModelThroughputCommand({ provisionedModelId: arn }));

What this demonstrates:

• Provisioning is a bedrock control-plane operation; running prompts stays on bedrock-runtime.
• modelUnits is the amount of reserved throughput; more units cost more and serve more.
• Once InService, you invoke by passing the provisionedModelArn as modelId - the inference code is unchanged.
• Deleting the provisioned model is what stops the hourly charge, so always clean up.

Deep Dive

On-Demand

On-demand is the default serving mode. You call a base model id, share pooled capacity with other tenants, and pay per input and output token. There is no reservation and nothing to tear down, which is why it suits development and variable traffic. The trade-offs are that throughput is subject to account-level request quotas and can be throttled under contention, and per-token latency is not guaranteed. Modern on-demand access to some models also runs through cross-region inference profiles (ids like us.anthropic.claude-...) rather than a bare model id.

Provisioned Throughput

Provisioned Throughput reserves dedicated capacity for a specific model, measured in model units, each providing a defined level of tokens-per-minute. Because the capacity is yours, latency is steadier and throughput is guaranteed up to what you bought - valuable for high, sustained production load. It is billed hourly, frequently with a commitment term (for example, monthly or longer), and no-commitment hourly options are more expensive per hour. Serving a customized (fine-tuned) model generally requires Provisioned Throughput as well.

Choosing Between Them

The decision is about volume, steadiness, and latency guarantees, not features - the API surface is identical.

A common path: build and launch on-demand, watch real traffic and throttling, and switch to provisioned only once sustained volume makes the reserved capacity cheaper and the latency guarantee worthwhile.

Gotchas

• Creating provisioned throughput to "try it" - it bills from creation regardless of use. Fix: develop on-demand; only provision for real sustained load, and delete when done.
• Calling Create on the wrong client - provisioning is a bedrock control-plane op, not bedrock-runtime. Fix: use the bedrock client to create, the runtime client to invoke.
• Invoking before it is InService - a freshly created provisioned model is not immediately ready. Fix: poll GetProvisionedModelThroughput until status is InService.
• Forgetting to delete - idle provisioned units keep charging. Fix: DeleteProvisionedModelThroughput and audit with ListProvisionedModelThroughputs.
• Expecting on-demand to never throttle - shared capacity is quota-limited. Fix: implement retries with backoff, or provision if you need guaranteed headroom.
• Assuming a bare model id always works on-demand - some models require an inference profile. Fix: list ListInferenceProfiles and use the profile id where required.

Alternatives

FAQs

Related

• Bedrock Basics - the on-demand Converse calls this builds on.
• Model Selection: Claude, Llama, Nova & Titan Trade-offs - capability side of the capacity decision.
• The Converse API vs InvokeModel - both surfaces work with either capacity model.
• Bedrock via SDK Best Practices - retries, quotas, and cost monitoring.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+). Bedrock model IDs and inference profiles are region/version-specific - verify against the current Bedrock catalog at build time.