Bedrock via SDK Best Practices

Prompt/response handling, retries, and token-cost monitoring.

This is the checklist to run before and after putting Bedrock inference into production. Each item is a rule stated positively with a one-line rationale, grouped from API choice outward through prompt handling, reliability, cost, and security. Work top to bottom - the early groups keep your code correct and portable; the later ones keep it cheap and safe.

How to Use This List

• Treat each box as an audit question against your own Bedrock integration.
• Groups run roughly in order of foundation: get the API and model choice right first, then handling, reliability, cost, and security.
• Most rules apply whether you use Converse or InvokeModel, on-demand or provisioned.
• Revisit the list whenever you add a model, a modality, or tool use.

A - API Choice & Model IDs

• Default to the Converse API. Its model-agnostic shape (messages, system, inferenceConfig, toolConfig) keeps code portable across families; reserve InvokeModel for embeddings, image generation, or model-specific params.
• Never hardcode a dated model id. IDs are region/version-specific and change; read them from config and list current options with ListFoundationModels.
• Use inference profiles where required. Modern on-demand access often needs a cross-region profile id (us.anthropic.claude-...); check ListInferenceProfiles.
• Confirm model access is enabled. Access errors usually mean the model is not enabled for your account/region, not an IAM bug - enable it in the Bedrock console.
• Verify feature support per model. Tools, vision, and documents are per-model; do not assume a family supports a capability you rely on.

B - Prompt & Response Handling

• Always set maxTokens. It caps reply length and output-token cost and prevents runaway generations.
• Keep instructions in system, not the user turn. The separate system field is treated as guidance and keeps prompts clean and reusable.
• Branch on stopReason. Handle max_tokens (truncated - raise the cap or continue), content_filtered (blocked), and tool_use (run a tool) distinctly from end_turn.
• Resend full conversation history deliberately. Bedrock is stateless per call; append the returned assistant message verbatim so roles and tool ids line up.
• Tune temperature for the task. Low for extraction and classification, higher for creative text; do not leave it at a default you have not considered.

C - Reliability & Retries

• Handle throttling with backoff. On-demand is quota-limited; configure the SDK's adaptive/standard retry mode and add jittered exponential backoff for ThrottlingException.
• Set sensible client timeouts. Long generations need generous read timeouts; too-short timeouts cause spurious failures and wasted retried tokens.
• Handle mid-stream errors. With ConverseStream, wrap iteration in try/except (boto3) or try/catch (v3) - errors can arrive during the stream, not just at the start.
• Cap the tool-use loop. A model can chain tool calls; loop until stopReason is end_turn with a maximum iteration count to avoid runaway cost.
• Make retries idempotent-aware. Retrying a generation re-bills tokens; retry on transient errors only, not on validation errors.

Configure retries and log the identifiers you will need to debug.

# --- Python (boto3) ---
import boto3
from botocore.config import Config
 
cfg = Config(
    retries={"max_attempts": 5, "mode": "adaptive"},  # backoff + throttling-aware
    read_timeout=120,                                  # long generations need headroom
)
brt = boto3.client("bedrock-runtime", region_name="us-east-1", config=cfg)
 
resp = brt.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=[{"role": "user", "content": [{"text": "Hi"}]}],
    inferenceConfig={"maxTokens": 200},
)
u = resp["usage"]
print("stop:", resp["stopReason"], "| tokens:", u["inputTokens"], u["outputTokens"])

// --- TypeScript (AWS SDK v3) ---
import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";
 
const brt = new BedrockRuntimeClient({
  region: "us-east-1",
  maxAttempts: 5,                 // retries with backoff
  requestHandler: { requestTimeout: 120_000 }, // long generations need headroom
});
 
const resp = await brt.send(new ConverseCommand({
  modelId: "amazon.nova-lite-v1:0",
  messages: [{ role: "user", content: [{ text: "Hi" }] }],
  inferenceConfig: { maxTokens: 200 },
}));
const u = resp.usage!;
console.log("stop:", resp.stopReason, "| tokens:", u.inputTokens, u.outputTokens);

D - Cost & Token Monitoring

• Log the usage block on every call. inputTokens/outputTokens are the basis for billing - emit them as metrics to catch cost regressions early.
• Right-size the model. Use the smallest tier that passes your evals; a top-tier model on a classification task is wasted spend.
• Trim conversation history. Old turns are re-billed as input tokens every call; summarize or drop stale context.
• Watch multimodal payload size. Images and documents inflate input tokens; downscale images and split large files.
• Choose the right capacity model. Stay on-demand for dev and spiky traffic; move to Provisioned Throughput only for sustained high volume, and delete idle provisioned units.
• Set budgets and cost alarms. A looping agent or unbounded generation can spend fast; surface it before the invoice does.

E - Security & IAM

• Grant least-privilege bedrock:InvokeModel. Scope the action to the specific model ARNs your workload calls, not *; add InvokeModelWithResponseStream only if you stream.
• Use roles, not access keys. On EC2, ECS, and Lambda, attach a role and let the SDK resolve rotating temporary credentials.
• Do not log sensitive prompt/response content. Prompts and outputs may contain PII; log token counts and request ids, not raw text, unless you have a data-handling reason.
• Consider Guardrails for safety. Apply Bedrock Guardrails to filter unsafe or off-policy content and redact sensitive data at the platform layer.
• Capture the request id on failures. Log $metadata.requestId (v3) or the boto3 response metadata so AWS support can trace calls.
• Separate control-plane permissions. Managing Provisioned Throughput uses bedrock control-plane actions; grant those only to the roles that provision, not to inference workloads.

FAQs

Related

• Bedrock's Unified Model-Invocation API - the API-choice rules in group A.
• Bedrock Basics - the prompt/response handling in group B.
• Streaming Responses with ConverseStream - the mid-stream error handling in group C.
• Provisioned Throughput & On-Demand Inference - the capacity choices in group D.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+). Bedrock model IDs and inference profiles are region/version-specific - verify against the current Bedrock catalog at build time.