AI Infrastructure via SDK Best Practices

Q: How do I find orphaned accelerator instances?

Filter DescribeInstances by tag and LaunchTime on a schedule, the same way you would sweep for any orphaned EC2 resource, and flag anything past its expected use window.

Accelerator instances - GPU, Trainium, Inferentia - are the most expensive compute most AWS accounts provision, and the easiest to leak money on: an idle p5.48xlarge left running, a Trainium cluster launched with no plan for cleanup, a training run blocked because nobody checked capacity or quota ahead of time.

These are the SDK-level habits that keep accelerator spend and failed launches under control. Review this list before shipping any code that provisions GPU, Trainium, or Inferentia capacity.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run in order of impact: planning and quota first, then provisioning, cost, and cleanup.
Check the boxes against your own scripts as a quick audit before a training run or serving deployment.
Revisit whenever you move to a new instance family, region, or account.

A - Planning & Quota

Request quota increases ahead of time. Large GPU, Trainium, and Inferentia instance families often start at a 0 vCPU quota per account; request the increase days before a planned launch, not the same day.
Check instance type offerings per AZ before launching. Not every accelerator type is offered in every Availability Zone - confirm with DescribeInstanceTypeOfferings first.
Reserve scarce capacity ahead of a scheduled run. Use a Capacity Reservation or a Capacity Block for ML instead of assuming on-demand capacity will be available on the day you need it.
Validate Neuron compatibility on one small instance first. Confirm a model compiles and runs correctly on Trainium/Inferentia before committing to a multi-node fleet.

Check offerings before committing to an AZ.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_type_offerings(
    LocationType="availability-zone",
    Filters=[{"Name": "instance-type", "Values": ["trn1.32xlarge"]}],
)
azs = [o["Location"] for o in resp["InstanceTypeOfferings"]]
print(azs)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypeOfferingsCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypeOfferingsCommand({
  LocationType: "availability-zone",
  Filters: [{ Name: "instance-type", Values: ["trn1.32xlarge"] }],
}));
const azs = (resp.InstanceTypeOfferings ?? []).map((o) => o.Location);
console.log(azs);

B - Provisioning & Networking

Provision from an AMI with drivers preinstalled. Use a Deep Learning AMI with GPU drivers or the Neuron SDK already set up rather than installing them at boot.
Launch multi-node clusters in one RunInstances call. A single call against a cluster placement group gives EC2 the best chance of packing nodes network-close.
Use EFA-enabled network interfaces for distributed training. Standard networking is not enough for tightly-coupled multi-node training; request InterfaceType: "efa" on supported instance types.
Tag every accelerator instance at launch. Use TagSpecifications on RunInstances, exactly as for any EC2 instance, so nothing expensive goes untracked.

Launch a cluster in one call, tagged, into a placement group.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0deeplearning-gpu",
    InstanceType="p5.48xlarge",
    MinCount=4, MaxCount=4,
    Placement={"GroupName": "training-cluster"},
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Environment", "Value": "training"},
                  {"Key": "Owner", "Value": "ml-team"}],
    }],
)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0deeplearning-gpu",
  InstanceType: "p5.48xlarge",
  MinCount: 4, MaxCount: 4,
  Placement: { GroupName: "training-cluster" },
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Environment", Value: "training" },
           { Key: "Owner", Value: "ml-team" }],
  }],
}));

C - Cost Control

Stop or terminate idle accelerator instances aggressively. These are the largest per-hour cost most AI infrastructure accounts carry; do not leave them running "just in case."
Remember storage keeps billing after a stop. Large model artifact volumes attached to a stopped GPU or Trainium instance keep billing for storage even while compute charges stop.
Use Spot for interruptible or checkpointed workloads. Batch training with regular checkpoints can tolerate Spot interruption at a significant discount over on-demand.
Release Capacity Reservations you no longer need. A reservation bills whether or not you launch into it; cancel it once the project or training run concludes.
Right-size before scaling up. Validate a workload on the smallest instance in a family (g5/inf2.xlarge/trn1.2xlarge) before committing to the largest.

D - Reliability & Cleanup

Wait on the full node list, not just the first instance. A training job should not start until every node in the cluster reaches running.
Widen waiter timeouts for accelerator instances. GPU and custom-silicon instances can take longer to boot than small general-purpose types.
Sweep for orphaned accelerator instances on a schedule. Filter by tag and launch time to find GPU/Trainium/Inferentia instances nobody is actively using.
Confirm accelerator readiness beyond instance_running. instance_running confirms the EC2 layer only; check driver, Neuron runtime, or framework health before starting a job.

Wait on the whole cluster before starting work.

# --- Python (boto3) ---
ec2.get_waiter("instance_running").wait(
    InstanceIds=node_ids,
    WaiterConfig={"Delay": 15, "MaxAttempts": 60},
)

// --- TypeScript (AWS SDK v3) ---
import { waitUntilInstanceRunning } from "@aws-sdk/client-ec2";
 
await waitUntilInstanceRunning(
  { client: ec2, maxWaitTime: 900 },
  { InstanceIds: nodeIds },
);

FAQs

Why request a quota increase before launching accelerator instances?

Large GPU, Trainium, and Inferentia families typically start at a 0 vCPU service quota per account; the increase can take time to approve, so requesting it early avoids blocking a planned launch.

What is the biggest cost risk with accelerator infrastructure?

Idle instances left running. GPU and custom-silicon instance-hours are the largest per-resource cost in most AI infrastructure accounts, far outweighing storage or networking.

Should I reserve capacity for every training run?

Only when the instance type is genuinely scarce or the run has a hard schedule. For flexible, non-scarce workloads, plain on-demand or Spot is simpler and cheaper.

Why validate Neuron compilation on one instance first?

Because discovering an unsupported model or op on a multi-node cluster is far more expensive, in both time and cost, than discovering it on a single small instance.

Does stopping a GPU instance stop all its charges?

No. Compute charges stop, but attached EBS storage - often large for model checkpoints and datasets - keeps billing while the instance is stopped.

Why does distributed training need EFA?

Standard networking does not provide the low latency and high bandwidth that tightly-coupled multi-node training's collective communication needs; EFA-enabled interfaces do.

How do I find orphaned accelerator instances?

Filter DescribeInstances by tag and LaunchTime on a schedule, the same way you would sweep for any orphaned EC2 resource, and flag anything past its expected use window.

Is instance_running enough to confirm an accelerator is ready?

No. It confirms the EC2 layer only. GPU drivers, the Neuron runtime, or your framework's own node-discovery step still need to finish inside the instance afterward.

Choosing AI Compute: GPUs vs Trainium/Inferentia - the accelerator-family decision behind group A.
P5/P6 GPU Instances for Large-Scale Training - the cluster placement group and EFA pattern in group B.
Capacity Reservations & Capacity Blocks for ML - the reservation tooling behind groups A and C.
AI Infrastructure Basics - the single-instance launch and waiter patterns behind group D.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

AI Infrastructure via SDK Best Practices

These are the SDK-level habits that keep accelerator spend and failed launches under control. Review this list before shipping any code that provisions GPU, Trainium, or Inferentia capacity.

How to Use This List

Each item is a rule stated positively, with a one-line rationale.
Groups run in order of impact: planning and quota first, then provisioning, cost, and cleanup.
Check the boxes against your own scripts as a quick audit before a training run or serving deployment.
Revisit whenever you move to a new instance family, region, or account.

A - Planning & Quota

Request quota increases ahead of time. Large GPU, Trainium, and Inferentia instance families often start at a 0 vCPU quota per account; request the increase days before a planned launch, not the same day.
Check instance type offerings per AZ before launching. Not every accelerator type is offered in every Availability Zone - confirm with DescribeInstanceTypeOfferings first.
Reserve scarce capacity ahead of a scheduled run. Use a Capacity Reservation or a Capacity Block for ML instead of assuming on-demand capacity will be available on the day you need it.
Validate Neuron compatibility on one small instance first. Confirm a model compiles and runs correctly on Trainium/Inferentia before committing to a multi-node fleet.

Check offerings before committing to an AZ.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.describe_instance_type_offerings(
    LocationType="availability-zone",
    Filters=[{"Name": "instance-type", "Values": ["trn1.32xlarge"]}],
)
azs = [o["Location"] for o in resp["InstanceTypeOfferings"]]
print(azs)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, DescribeInstanceTypeOfferingsCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
const resp = await ec2.send(new DescribeInstanceTypeOfferingsCommand({
  LocationType: "availability-zone",
  Filters: [{ Name: "instance-type", Values: ["trn1.32xlarge"] }],
}));
const azs = (resp.InstanceTypeOfferings ?? []).map((o) => o.Location);
console.log(azs);

B - Provisioning & Networking

Provision from an AMI with drivers preinstalled. Use a Deep Learning AMI with GPU drivers or the Neuron SDK already set up rather than installing them at boot.
Launch multi-node clusters in one RunInstances call. A single call against a cluster placement group gives EC2 the best chance of packing nodes network-close.
Use EFA-enabled network interfaces for distributed training. Standard networking is not enough for tightly-coupled multi-node training; request InterfaceType: "efa" on supported instance types.
Tag every accelerator instance at launch. Use TagSpecifications on RunInstances, exactly as for any EC2 instance, so nothing expensive goes untracked.

Launch a cluster in one call, tagged, into a placement group.

# --- Python (boto3) ---
import boto3
 
ec2 = boto3.client("ec2", region_name="us-east-1")
resp = ec2.run_instances(
    ImageId="ami-0deeplearning-gpu",
    InstanceType="p5.48xlarge",
    MinCount=4, MaxCount=4,
    Placement={"GroupName": "training-cluster"},
    TagSpecifications=[{
        "ResourceType": "instance",
        "Tags": [{"Key": "Environment", "Value": "training"},
                  {"Key": "Owner", "Value": "ml-team"}],
    }],
)

// --- TypeScript (AWS SDK v3) ---
import { EC2Client, RunInstancesCommand } from "@aws-sdk/client-ec2";
 
const ec2 = new EC2Client({ region: "us-east-1" });
await ec2.send(new RunInstancesCommand({
  ImageId: "ami-0deeplearning-gpu",
  InstanceType: "p5.48xlarge",
  MinCount: 4, MaxCount: 4,
  Placement: { GroupName: "training-cluster" },
  TagSpecifications: [{
    ResourceType: "instance",
    Tags: [{ Key: "Environment", Value: "training" },
           { Key: "Owner", Value: "ml-team" }],
  }],
}));

C - Cost Control

Stop or terminate idle accelerator instances aggressively. These are the largest per-hour cost most AI infrastructure accounts carry; do not leave them running "just in case."
Remember storage keeps billing after a stop. Large model artifact volumes attached to a stopped GPU or Trainium instance keep billing for storage even while compute charges stop.
Use Spot for interruptible or checkpointed workloads. Batch training with regular checkpoints can tolerate Spot interruption at a significant discount over on-demand.
Release Capacity Reservations you no longer need. A reservation bills whether or not you launch into it; cancel it once the project or training run concludes.
Right-size before scaling up. Validate a workload on the smallest instance in a family (g5/inf2.xlarge/trn1.2xlarge) before committing to the largest.

D - Reliability & Cleanup

Wait on the full node list, not just the first instance. A training job should not start until every node in the cluster reaches running.
Widen waiter timeouts for accelerator instances. GPU and custom-silicon instances can take longer to boot than small general-purpose types.
Sweep for orphaned accelerator instances on a schedule. Filter by tag and launch time to find GPU/Trainium/Inferentia instances nobody is actively using.
Confirm accelerator readiness beyond instance_running. instance_running confirms the EC2 layer only; check driver, Neuron runtime, or framework health before starting a job.

Wait on the whole cluster before starting work.

# --- Python (boto3) ---
ec2.get_waiter("instance_running").wait(
    InstanceIds=node_ids,
    WaiterConfig={"Delay": 15, "MaxAttempts": 60},
)

// --- TypeScript (AWS SDK v3) ---
import { waitUntilInstanceRunning } from "@aws-sdk/client-ec2";
 
await waitUntilInstanceRunning(
  { client: ec2, maxWaitTime: 900 },
  { InstanceIds: nodeIds },
);

FAQs

Why request a quota increase before launching accelerator instances?

Large GPU, Trainium, and Inferentia families typically start at a 0 vCPU service quota per account; the increase can take time to approve, so requesting it early avoids blocking a planned launch.

What is the biggest cost risk with accelerator infrastructure?

Idle instances left running. GPU and custom-silicon instance-hours are the largest per-resource cost in most AI infrastructure accounts, far outweighing storage or networking.

Should I reserve capacity for every training run?

Only when the instance type is genuinely scarce or the run has a hard schedule. For flexible, non-scarce workloads, plain on-demand or Spot is simpler and cheaper.

Why validate Neuron compilation on one instance first?

Because discovering an unsupported model or op on a multi-node cluster is far more expensive, in both time and cost, than discovering it on a single small instance.

Does stopping a GPU instance stop all its charges?

No. Compute charges stop, but attached EBS storage - often large for model checkpoints and datasets - keeps billing while the instance is stopped.

Why does distributed training need EFA?

Standard networking does not provide the low latency and high bandwidth that tightly-coupled multi-node training's collective communication needs; EFA-enabled interfaces do.

How do I find orphaned accelerator instances?

Filter DescribeInstances by tag and LaunchTime on a schedule, the same way you would sweep for any orphaned EC2 resource, and flag anything past its expected use window.

Is instance_running enough to confirm an accelerator is ready?

No. It confirms the EC2 layer only. GPU drivers, the Neuron runtime, or your framework's own node-discovery step still need to finish inside the instance afterward.

Choosing AI Compute: GPUs vs Trainium/Inferentia - the accelerator-family decision behind group A.
P5/P6 GPU Instances for Large-Scale Training - the cluster placement group and EFA pattern in group B.
Capacity Reservations & Capacity Blocks for ML - the reservation tooling behind groups A and C.
AI Infrastructure Basics - the single-instance launch and waiter patterns behind group D.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

AI Infrastructure via SDK Best Practices

Buscar en la documentación

AI Infrastructure via SDK Best Practices