AI Infrastructure Basics

Seven short examples for provisioning accelerator instances - GPU, Trainium, and Inferentia - through the same EC2 SDK calls you would use for any other instance type.

Each example shows Python (boto3) and TypeScript (AWS SDK for JavaScript v3) side by side, so the mechanics translate one to one.

Prerequisites

Python: pip install boto3 (boto3 1.43.x, Python 3.10+). TypeScript: npm install @aws-sdk/client-ec2 (Node.js 18+).
Credentials configured via aws configure, environment variables, or an IAM role with ec2:RunInstances, ec2:Describe*, and ec2:CreateTags.
Accelerator instance types (p5, trn1, inf2, etc.) require quota approval in most accounts before you can launch them; request a service quota increase first.

Basic Examples

1. Check Instance Type Availability in an AZ

Not every accelerator instance type is offered in every Availability Zone. Check before you try to launch.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_type_offerings(
    LocationType="availability-zone",
    Filters=[{"Name": "instance-type", "Values": ["trn1.32xlarge"]}],
)
for offering in resp["InstanceTypeOfferings"]:
    print(offering["Location"], offering["InstanceType"])

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypeOfferingsCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypeOfferingsCommand({
  LocationType: "availability-zone",
  Filters: [{ Name: "instance-type", Values: ["trn1.32xlarge"] }],
}));
for (const offering of resp.InstanceTypeOfferings ?? []) {
  console.log(offering.Location, offering.InstanceType);
}

An empty result means that instance type is not offered in that region at all.
A non-empty result lists the specific AZs where the type is offered - launch attempts outside those AZs fail.
This is a read-only, free API call worth running before every new accelerator launch script.
The same call works for p5, p6, g5, g6, inf1, and inf2 types by changing the filter value.

2. Launch a Single GPU Instance

RunInstances with a GPU instance type looks exactly like launching any other EC2 instance.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0deeplearning",   # a Deep Learning AMI with GPU drivers preinstalled
    InstanceType="g5.2xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "gpu-dev"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]
print(instance_id)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0deeplearning",   // a Deep Learning AMI with GPU drivers preinstalled
  InstanceType: "g5.2xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "gpu-dev" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;
console.log(instanceId);

Use an AMI with GPU drivers and a CUDA runtime preinstalled (an AWS Deep Learning AMI) rather than a bare Linux AMI.
g5.2xlarge is a smaller-scale GPU type suited to development and light inference; larger training uses p5/p6.
Tag at launch, exactly as covered in the EC2 section, so idle GPU capacity is easy to find.
The response returns the instance in pending state; it is not usable yet.

3. Launch a Trainium Instance

Trainium instances launch the same way; only the instance type and AMI differ.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0neuron",   # AMI with the AWS Neuron SDK preinstalled
    InstanceType="trn1.32xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "trainium-train"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0neuron",   // AMI with the AWS Neuron SDK preinstalled
  InstanceType: "trn1.32xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "trainium-train" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;

Use an AMI with the Neuron SDK and Neuron-compatible framework packages preinstalled.
trn1.32xlarge is a large multi-accelerator instance; smaller Trainium sizes exist for lighter workloads.
Everything about the launch call - tags, security groups, subnet - works the same as any EC2 instance.
Your model still needs to be compiled through Neuron before training will actually use the accelerator.

Related: Trainium for Cost-Efficient Training - the Neuron compilation step in full.

4. Launch an Inferentia Instance for Serving

Inferentia instances follow the identical pattern, sized for inference serving.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0neuron-inf",
    InstanceType="inf2.xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "inference-serve"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0neuron-inf",
  InstanceType: "inf2.xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "inference-serve" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;

inf2.xlarge is the smallest Inferentia 2 size, good for lower-throughput serving or testing a Neuron-compiled model.
Larger inf2 sizes add more accelerators for higher-throughput serving behind a load balancer.
The launch call itself carries no inference-specific parameters; serving logic lives in your application, not the SDK.

Intermediate Examples

5. Wait for the Instance to Be Ready

Use the standard instance_running waiter before assuming the instance can be used - accelerator instances often take longer to reach running than a small general-purpose instance.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
ec2.get_waiter("instance_running").wait(
    InstanceIds=[instance_id],
    WaiterConfig={"Delay": 15, "MaxAttempts": 40},
)
print("instance is running")

// --- TypeScript (AWS SDK v3) ---
import { waitUntilInstanceRunning } from "@aws-sdk/client-ec2";
 
await waitUntilInstanceRunning(
  { client: ec2, maxWaitTime: 600 },
  { InstanceIds: [instanceId] },
);
console.log("instance is running");

Accelerator instances can take longer to boot than small general-purpose types; widen the waiter's max attempts or wait time accordingly.
instance_running only confirms the EC2 layer is up, not that GPU drivers or the Neuron runtime finished initializing.
Pair this with a health check inside your bootstrap script if you need to confirm the accelerator itself is ready.

6. Describe Accelerator Details on a Running Instance

DescribeInstanceTypes reports the accelerator count and type for a given instance type, useful for validating a launch programmatically.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_types(InstanceTypes=["p5.48xlarge"])
info = resp["InstanceTypes"][0]
print(info.get("GpuInfo") or info.get("AcceleratorInfo"))

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypesCommand({
  InstanceTypes: ["p5.48xlarge"],
}));
const info = resp.InstanceTypes?.[0];
console.log(info?.GpuInfo ?? info?.NeuronInfo);

GPU instance types report accelerator details under GpuInfo; Trainium/Inferentia types report them under a Neuron-specific field.
This call is metadata only - it describes the instance type, not a specific running instance.
Useful in automation that picks an instance type based on required accelerator count rather than a hardcoded string.

7. Stop an Idle Accelerator Instance

Accelerator instances are expensive per hour; stop them the same way as any EC2 instance when idle.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
ec2.stop_instances(InstanceIds=[instance_id])
ec2.get_waiter("instance_stopped").wait(InstanceIds=[instance_id])
print("stopped")

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, StopInstancesCommand, waitUntilInstanceStopped } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
await ec2.send(new StopInstancesCommand({ InstanceIds: [instanceId] }));
await waitUntilInstanceStopped({ client: ec2, maxWaitTime: 300 }, { InstanceIds: [instanceId] });
console.log("stopped");

Stopping halts the accelerator compute charge, but attached EBS volumes (often large for model artifacts) keep billing.
For large P5/P6 or Trainium clusters, prefer scripted stop/terminate over leaving them running "just in case."
If capacity for the instance type is scarce, consider a Capacity Reservation before stopping, so restarting later does not mean losing your spot.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Each example shows Python (boto3) and TypeScript (AWS SDK for JavaScript v3) side by side, so the mechanics translate one to one.

Prerequisites

Python: pip install boto3 (boto3 1.43.x, Python 3.10+). TypeScript: npm install @aws-sdk/client-ec2 (Node.js 18+).
Credentials configured via aws configure, environment variables, or an IAM role with ec2:RunInstances, ec2:Describe*, and ec2:CreateTags.
Accelerator instance types (p5, trn1, inf2, etc.) require quota approval in most accounts before you can launch them; request a service quota increase first.

Basic Examples

1. Check Instance Type Availability in an AZ

Not every accelerator instance type is offered in every Availability Zone. Check before you try to launch.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_type_offerings(
    LocationType="availability-zone",
    Filters=[{"Name": "instance-type", "Values": ["trn1.32xlarge"]}],
)
for offering in resp["InstanceTypeOfferings"]:
    print(offering["Location"], offering["InstanceType"])

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypeOfferingsCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypeOfferingsCommand({
  LocationType: "availability-zone",
  Filters: [{ Name: "instance-type", Values: ["trn1.32xlarge"] }],
}));
for (const offering of resp.InstanceTypeOfferings ?? []) {
  console.log(offering.Location, offering.InstanceType);
}

An empty result means that instance type is not offered in that region at all.
A non-empty result lists the specific AZs where the type is offered - launch attempts outside those AZs fail.
This is a read-only, free API call worth running before every new accelerator launch script.
The same call works for p5, p6, g5, g6, inf1, and inf2 types by changing the filter value.

2. Launch a Single GPU Instance

RunInstances with a GPU instance type looks exactly like launching any other EC2 instance.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0deeplearning",   # a Deep Learning AMI with GPU drivers preinstalled
    InstanceType="g5.2xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "gpu-dev"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]
print(instance_id)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0deeplearning",   // a Deep Learning AMI with GPU drivers preinstalled
  InstanceType: "g5.2xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "gpu-dev" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;
console.log(instanceId);

Use an AMI with GPU drivers and a CUDA runtime preinstalled (an AWS Deep Learning AMI) rather than a bare Linux AMI.
g5.2xlarge is a smaller-scale GPU type suited to development and light inference; larger training uses p5/p6.
Tag at launch, exactly as covered in the EC2 section, so idle GPU capacity is easy to find.
The response returns the instance in pending state; it is not usable yet.

3. Launch a Trainium Instance

Trainium instances launch the same way; only the instance type and AMI differ.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0neuron",   # AMI with the AWS Neuron SDK preinstalled
    InstanceType="trn1.32xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "trainium-train"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0neuron",   // AMI with the AWS Neuron SDK preinstalled
  InstanceType: "trn1.32xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "trainium-train" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;

Use an AMI with the Neuron SDK and Neuron-compatible framework packages preinstalled.
trn1.32xlarge is a large multi-accelerator instance; smaller Trainium sizes exist for lighter workloads.
Everything about the launch call - tags, security groups, subnet - works the same as any EC2 instance.
Your model still needs to be compiled through Neuron before training will actually use the accelerator.

Related: Trainium for Cost-Efficient Training - the Neuron compilation step in full.

4. Launch an Inferentia Instance for Serving

Inferentia instances follow the identical pattern, sized for inference serving.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0neuron-inf",
    InstanceType="inf2.xlarge",
    MinCount=1, MaxCount=1,
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Name", "Value": "inference-serve"}],
    }],
)
instance_id = resp["Instances"][0]["InstanceId"]

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0neuron-inf",
  InstanceType: "inf2.xlarge",
  MinCount: 1, MaxCount: 1,
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Name", Value: "inference-serve" }],
  }],
}));
const instanceId = resp.Instances?.[0].InstanceId;

inf2.xlarge is the smallest Inferentia 2 size, good for lower-throughput serving or testing a Neuron-compiled model.
Larger inf2 sizes add more accelerators for higher-throughput serving behind a load balancer.
The launch call itself carries no inference-specific parameters; serving logic lives in your application, not the SDK.

Intermediate Examples

5. Wait for the Instance to Be Ready

Use the standard instance_running waiter before assuming the instance can be used - accelerator instances often take longer to reach running than a small general-purpose instance.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
ec2.get_waiter("instance_running").wait(
    InstanceIds=[instance_id],
    WaiterConfig={"Delay": 15, "MaxAttempts": 40},
)
print("instance is running")

// --- TypeScript (AWS SDK v3) ---
import { waitUntilInstanceRunning } from "@aws-sdk/client-ec2";
 
await waitUntilInstanceRunning(
  { client: ec2, maxWaitTime: 600 },
  { InstanceIds: [instanceId] },
);
console.log("instance is running");

Accelerator instances can take longer to boot than small general-purpose types; widen the waiter's max attempts or wait time accordingly.
instance_running only confirms the EC2 layer is up, not that GPU drivers or the Neuron runtime finished initializing.
Pair this with a health check inside your bootstrap script if you need to confirm the accelerator itself is ready.

6. Describe Accelerator Details on a Running Instance

DescribeInstanceTypes reports the accelerator count and type for a given instance type, useful for validating a launch programmatically.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_types(InstanceTypes=["p5.48xlarge"])
info = resp["InstanceTypes"][0]
print(info.get("GpuInfo") or info.get("AcceleratorInfo"))

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypesCommand({
  InstanceTypes: ["p5.48xlarge"],
}));
const info = resp.InstanceTypes?.[0];
console.log(info?.GpuInfo ?? info?.NeuronInfo);

GPU instance types report accelerator details under GpuInfo; Trainium/Inferentia types report them under a Neuron-specific field.
This call is metadata only - it describes the instance type, not a specific running instance.
Useful in automation that picks an instance type based on required accelerator count rather than a hardcoded string.

7. Stop an Idle Accelerator Instance

Accelerator instances are expensive per hour; stop them the same way as any EC2 instance when idle.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
ec2.stop_instances(InstanceIds=[instance_id])
ec2.get_waiter("instance_stopped").wait(InstanceIds=[instance_id])
print("stopped")

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, StopInstancesCommand, waitUntilInstanceStopped } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
await ec2.send(new StopInstancesCommand({ InstanceIds: [instanceId] }));
await waitUntilInstanceStopped({ client: ec2, maxWaitTime: 300 }, { InstanceIds: [instanceId] });
console.log("stopped");

Stopping halts the accelerator compute charge, but attached EBS volumes (often large for model artifacts) keep billing.
For large P5/P6 or Trainium clusters, prefer scripted stop/terminate over leaving them running "just in case."
If capacity for the instance type is scarce, consider a Capacity Reservation before stopping, so restarting later does not mean losing your spot.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

AI Infrastructure Basics

Search Documentation

AI Infrastructure Basics