Experiment Tracking with SageMaker Experiments

Q: Do I need to call a separate logging API for basic training-job tracking?

No. Passing ExperimentConfig on CreateTrainingJob is enough - SageMaker creates and associates a trial component automatically, capturing the job's hyperparameters and any defined metrics.

Q: Can I log a custom metric that isn't produced by a training job?

Yes. Use CreateTrialComponent to create a component directly, then UpdateTrialComponent to add parameters or metrics, and AssociateTrialComponent to attach it to a trial.

Q: How do I compare many trials at once?

ListTrialComponents scoped to a trial or experiment returns the summaries to iterate and describe. For filtering by metric value across many components, the Search API accepts a filter expression instead of listing and checking each one yourself.

A handful of training jobs are easy to keep straight by name. A few dozen tuning trials, retrained weekly, across a team, are not - without structure you end up guessing which run used which data and which hyperparameters produced which result.

SageMaker Experiments gives training runs a hierarchy - Experiment > Trial > Trial Component - and a metadata store so you can compare runs later without re-deriving what each one did from job names alone.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
sm = boto3.client("sagemaker", region_name="us-east-1")
 
sm.create_experiment(ExperimentName="churn-model", Description="Customer churn model iterations")
sm.create_trial(TrialName="churn-trial-xgb-v1", ExperimentName="churn-model")
 
# Associate the training job with the trial - SageMaker creates the trial component for you.
sm.create_training_job(
    TrainingJobName="churn-xgb-v1",
    AlgorithmSpecification={"TrainingImage": "683313688378.dkr.ecr.us-east-1.amazonaws.com/xgboost:1.7-1",
                             "TrainingInputMode": "File"},
    RoleArn="arn:aws:iam::111122223333:role/SageMakerExecutionRole",
    InputDataConfig=[{"ChannelName": "train",
                       "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix",
                                                        "S3Uri": "s3://my-ml-bucket/train/"}}}],
    OutputDataConfig={"S3OutputPath": "s3://my-ml-bucket/output/"},
    ResourceConfig={"InstanceType": "ml.m5.xlarge", "InstanceCount": 1, "VolumeSizeInGB": 30},
    HyperParameters={"num_round": "100", "objective": "binary:logistic"},
    StoppingCondition={"MaxRuntimeInSeconds": 1800},
    ExperimentConfig={
        "ExperimentName": "churn-model",
        "TrialName": "churn-trial-xgb-v1",
        "TrialComponentDisplayName": "xgb-v1-training",
    },
)

// --- TypeScript (AWS SDK v3) ---
import {
  SageMakerClient, CreateExperimentCommand, CreateTrialCommand, CreateTrainingJobCommand,
} from "@aws-sdk/client-sagemaker";
 
const sm = new SageMakerClient({ region: "us-east-1" });
 
await sm.send(new CreateExperimentCommand({ ExperimentName: "churn-model", Description: "Customer churn model iterations" }));
await sm.send(new CreateTrialCommand({ TrialName: "churn-trial-xgb-v1", ExperimentName: "churn-model" }));
 
// Associate the training job with the trial - SageMaker creates the trial component for you.
await sm.send(new CreateTrainingJobCommand({
  TrainingJobName: "churn-xgb-v1",
  AlgorithmSpecification: { TrainingImage: "683313688378.dkr.ecr.us-east-1.amazonaws.com/xgboost:1.7-1",
                             TrainingInputMode: "File" },
  RoleArn: "arn:aws:iam::111122223333:role/SageMakerExecutionRole",
  InputDataConfig: [{ ChannelName: "train",
                       DataSource: { S3DataSource: { S3DataType: "S3Prefix",
                                                      S3Uri: "s3://my-ml-bucket/train/" } } }],
  OutputDataConfig: { S3OutputPath: "s3://my-ml-bucket/output/" },
  ResourceConfig: { InstanceType: "ml.m5.xlarge", InstanceCount: 1, VolumeSizeInGB: 30 },
  HyperParameters: { num_round: "100", objective: "binary:logistic" },
  StoppingCondition: { MaxRuntimeInSeconds: 1800 },
  ExperimentConfig: {
    ExperimentName: "churn-model",
    TrialName: "churn-trial-xgb-v1",
    TrialComponentDisplayName: "xgb-v1-training",
  },
}));

When to reach for this:

Running the same model idea repeatedly with different data, hyperparameters, or code versions.
Needing to answer "which run produced this model artifact, and with what settings?" months later.
Comparing tuning-job trials or ad hoc training jobs side by side without re-reading each DescribeTrainingJob individually.

Working Example

List the trial components in a trial and compare their logged parameters and metrics.

# --- Python (boto3) ---
components = sm.list_trial_components(TrialName="churn-trial-xgb-v1")["TrialComponentSummaries"]
for c in components:
    detail = sm.describe_trial_component(TrialComponentName=c["TrialComponentName"])
    print(detail["TrialComponentName"])
    print("  parameters:", detail.get("Parameters"))
    print("  metrics:", [(m["MetricName"], m.get("Avg")) for m in detail.get("Metrics", [])])

// --- TypeScript (AWS SDK v3) ---
import { ListTrialComponentsCommand, DescribeTrialComponentCommand } from "@aws-sdk/client-sagemaker";
 
const list = await sm.send(new ListTrialComponentsCommand({ TrialName: "churn-trial-xgb-v1" }));
for (const c of list.TrialComponentSummaries ?? []) {
  const detail = await sm.send(new DescribeTrialComponentCommand({ TrialComponentName: c.TrialComponentName }));
  console.log(detail.TrialComponentName);
  console.log("  parameters:", detail.Parameters);
  console.log("  metrics:", detail.Metrics?.map((m) => [m.MetricName, m.Avg]));
}

What this demonstrates:

The trial component created automatically by ExperimentConfig already carries the training job's hyperparameters and any metrics defined via MetricDefinitions.
Iterating ListTrialComponents across a trial (or an experiment) is how you build a comparison table without hand-tracking job names.
No separate logging call was needed for these basics - associating the job at creation time was enough.
Custom containers can log additional metrics into the same trial component with CreateTrialComponent/UpdateTrialComponent, covered below.

Deep Dive

The hierarchy: Experiment, Trial, Trial Component

An Experiment is the top-level container for a problem you're iterating on ("churn-model"). A Trial is one attempt within it - typically one full run of your workflow, which might be a single training job or a short pipeline of steps. A Trial Component is the individual step: a training job, a processing job, or a manually logged step, each carrying its own parameters, metrics, and input/output artifact references.

A tuning job's trials map naturally onto this: each training job launched by CreateHyperParameterTuningJob can be associated with its own trial component, letting you compare an entire search inside one experiment.

Automatic association via ExperimentConfig

The simplest path, shown in the recipe, is passing ExperimentConfig directly on CreateTrainingJob (and similarly on processing or transform jobs). SageMaker creates a trial component for that job automatically, populates it with the job's hyperparameters, and associates it with the named trial and experiment - no separate CreateTrialComponent call required for the training job itself.

Manual trial components for custom metadata

For metadata that doesn't come from a training job directly - a data-preprocessing step, a manual evaluation note, or a metric computed after the fact - use CreateTrialComponent directly, then AssociateTrialComponent to attach it to a trial. UpdateTrialComponent lets you add parameters or metrics to an existing component as they become available, such as after an offline evaluation finishes.

Comparing runs

Once components exist, ListTrialComponents (scoped by TrialName or ExperimentName) and DescribeTrialComponent give you the raw data to build a comparison. For larger-scale filtering - "find every trial component with validation:auc above some threshold" - the SageMaker Search API accepts a filter expression against experiment/trial/trial-component metadata and returns matches without listing everything yourself.

A note on the higher-level convenience API

The separate SageMaker Python SDK (the sagemaker package, distinct from boto3) ships a higher-level Run/experiments convenience layer that wraps these same Create*/Describe* calls with a more ergonomic context-manager style API. Its exact method names and behavior have shifted across SDK versions, so this page sticks to the underlying boto3/SDK v3 calls, which are stable across that layer's changes. If you use that convenience SDK, check its current documentation for the exact API rather than assuming it matches boto3's method names one-to-one.

Gotchas

Expecting ExperimentConfig to create the Experiment or Trial. It only associates a job with names that must already exist. Fix: call CreateExperiment and CreateTrial first.
Logging metrics but forgetting MetricDefinitions. Without a matching regex in AlgorithmSpecification.MetricDefinitions, no metric ever reaches the trial component. Fix: define metrics the same way you would for a tuning job's objective.
Assuming trial components store the data itself. They store metadata and references; the actual datasets and model artifacts still live in S3 where the job wrote them. Fix: treat the trial component as an index, not a data store.
Re-using a TrialComponentDisplayName across unrelated jobs. It becomes hard to tell components apart when comparing later. Fix: name components so they're identifiable without opening each one.
Confusing this SDK-level Experiments API with the higher-level Python SDK's Run helper. They cover the same underlying resources but not always the same method names. Fix: pick one layer for a given project and check its current docs rather than mixing assumptions.

Alternatives

Alternative	Use When	Don't Use When
SageMaker Experiments (this page)	You want run metadata to live alongside SageMaker jobs with no extra infrastructure	You already run a dedicated experiment-tracking tool your team standardized on
Higher-level SageMaker Python SDK Run API	You want a more ergonomic wrapper and are comfortable tracking its version-specific API	You need the most stable, version-independent surface (use boto3 directly)
Third-party experiment tracking (self-hosted or SaaS)	Your team already has one in place across non-SageMaker workloads too	You want to avoid running or paying for extra infrastructure
Naming convention + S3 tags only	A handful of jobs, low iteration frequency	More than a few runs, or multiple people iterating on the same problem

FAQs

Do I need to call a separate logging API for basic training-job tracking?

No. Passing ExperimentConfig on CreateTrainingJob is enough - SageMaker creates and associates a trial component automatically, capturing the job's hyperparameters and any defined metrics.

What's the difference between an Experiment, a Trial, and a Trial Component?

An Experiment groups everything related to one problem. A Trial is one attempt at it. A Trial Component is one step of that attempt - most often a single training job, but it can be any logged step.

Can I log a custom metric that isn't produced by a training job?

Yes. Use CreateTrialComponent to create a component directly, then UpdateTrialComponent to add parameters or metrics, and AssociateTrialComponent to attach it to a trial.

How do I compare many trials at once?

ListTrialComponents scoped to a trial or experiment returns the summaries to iterate and describe. For filtering by metric value across many components, the Search API accepts a filter expression instead of listing and checking each one yourself.

Does SageMaker Experiments store my training data or model files?

No. It stores metadata - parameters, metrics, and references to where artifacts live. The actual data and model artifacts remain in S3 exactly where the training job wrote them.

Is there a simpler API than these boto3 calls?

The separate, higher-level SageMaker Python SDK offers a convenience experiments/Run API on top of these same resources. Its exact surface has changed across versions, so check that SDK's current documentation if you plan to use it instead of boto3 directly.

SageMaker Training Basics - the training job this page attaches ExperimentConfig to.
Hyperparameter Tuning Jobs - a natural source of many trials to compare in one experiment.
Training Jobs: Built-in Algorithms vs Custom Containers - MetricDefinitions, shared with tuning and experiment tracking.
SageMaker Training via SDK Best Practices - reproducibility habits this page's tracking supports.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
sm = boto3.client("sagemaker", region_name="us-east-1")
 
sm.create_experiment(ExperimentName="churn-model", Description="Customer churn model iterations")
sm.create_trial(TrialName="churn-trial-xgb-v1", ExperimentName="churn-model")
 
# Associate the training job with the trial - SageMaker creates the trial component for you.
sm.create_training_job(
    TrainingJobName="churn-xgb-v1",
    AlgorithmSpecification={"TrainingImage": "683313688378.dkr.ecr.us-east-1.amazonaws.com/xgboost:1.7-1",
                             "TrainingInputMode": "File"},
    RoleArn="arn:aws:iam::111122223333:role/SageMakerExecutionRole",
    InputDataConfig=[{"ChannelName": "train",
                       "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix",
                                                        "S3Uri": "s3://my-ml-bucket/train/"}}}],
    OutputDataConfig={"S3OutputPath": "s3://my-ml-bucket/output/"},
    ResourceConfig={"InstanceType": "ml.m5.xlarge", "InstanceCount": 1, "VolumeSizeInGB": 30},
    HyperParameters={"num_round": "100", "objective": "binary:logistic"},
    StoppingCondition={"MaxRuntimeInSeconds": 1800},
    ExperimentConfig={
        "ExperimentName": "churn-model",
        "TrialName": "churn-trial-xgb-v1",
        "TrialComponentDisplayName": "xgb-v1-training",
    },
)

// --- TypeScript (AWS SDK v3) ---
import {
  SageMakerClient, CreateExperimentCommand, CreateTrialCommand, CreateTrainingJobCommand,
} from "@aws-sdk/client-sagemaker";
 
const sm = new SageMakerClient({ region: "us-east-1" });
 
await sm.send(new CreateExperimentCommand({ ExperimentName: "churn-model", Description: "Customer churn model iterations" }));
await sm.send(new CreateTrialCommand({ TrialName: "churn-trial-xgb-v1", ExperimentName: "churn-model" }));
 
// Associate the training job with the trial - SageMaker creates the trial component for you.
await sm.send(new CreateTrainingJobCommand({
  TrainingJobName: "churn-xgb-v1",
  AlgorithmSpecification: { TrainingImage: "683313688378.dkr.ecr.us-east-1.amazonaws.com/xgboost:1.7-1",
                             TrainingInputMode: "File" },
  RoleArn: "arn:aws:iam::111122223333:role/SageMakerExecutionRole",
  InputDataConfig: [{ ChannelName: "train",
                       DataSource: { S3DataSource: { S3DataType: "S3Prefix",
                                                      S3Uri: "s3://my-ml-bucket/train/" } } }],
  OutputDataConfig: { S3OutputPath: "s3://my-ml-bucket/output/" },
  ResourceConfig: { InstanceType: "ml.m5.xlarge", InstanceCount: 1, VolumeSizeInGB: 30 },
  HyperParameters: { num_round: "100", objective: "binary:logistic" },
  StoppingCondition: { MaxRuntimeInSeconds: 1800 },
  ExperimentConfig: {
    ExperimentName: "churn-model",
    TrialName: "churn-trial-xgb-v1",
    TrialComponentDisplayName: "xgb-v1-training",
  },
}));

When to reach for this:

Running the same model idea repeatedly with different data, hyperparameters, or code versions.
Needing to answer "which run produced this model artifact, and with what settings?" months later.
Comparing tuning-job trials or ad hoc training jobs side by side without re-reading each DescribeTrainingJob individually.

Working Example

List the trial components in a trial and compare their logged parameters and metrics.

# --- Python (boto3) ---
components = sm.list_trial_components(TrialName="churn-trial-xgb-v1")["TrialComponentSummaries"]
for c in components:
    detail = sm.describe_trial_component(TrialComponentName=c["TrialComponentName"])
    print(detail["TrialComponentName"])
    print("  parameters:", detail.get("Parameters"))
    print("  metrics:", [(m["MetricName"], m.get("Avg")) for m in detail.get("Metrics", [])])

// --- TypeScript (AWS SDK v3) ---
import { ListTrialComponentsCommand, DescribeTrialComponentCommand } from "@aws-sdk/client-sagemaker";
 
const list = await sm.send(new ListTrialComponentsCommand({ TrialName: "churn-trial-xgb-v1" }));
for (const c of list.TrialComponentSummaries ?? []) {
  const detail = await sm.send(new DescribeTrialComponentCommand({ TrialComponentName: c.TrialComponentName }));
  console.log(detail.TrialComponentName);
  console.log("  parameters:", detail.Parameters);
  console.log("  metrics:", detail.Metrics?.map((m) => [m.MetricName, m.Avg]));
}

What this demonstrates:

The trial component created automatically by ExperimentConfig already carries the training job's hyperparameters and any metrics defined via MetricDefinitions.
Iterating ListTrialComponents across a trial (or an experiment) is how you build a comparison table without hand-tracking job names.
No separate logging call was needed for these basics - associating the job at creation time was enough.
Custom containers can log additional metrics into the same trial component with CreateTrialComponent/UpdateTrialComponent, covered below.

Deep Dive

The hierarchy: Experiment, Trial, Trial Component

Automatic association via ExperimentConfig

Manual trial components for custom metadata

Comparing runs

A note on the higher-level convenience API

Gotchas

Expecting ExperimentConfig to create the Experiment or Trial. It only associates a job with names that must already exist. Fix: call CreateExperiment and CreateTrial first.
Logging metrics but forgetting MetricDefinitions. Without a matching regex in AlgorithmSpecification.MetricDefinitions, no metric ever reaches the trial component. Fix: define metrics the same way you would for a tuning job's objective.
Assuming trial components store the data itself. They store metadata and references; the actual datasets and model artifacts still live in S3 where the job wrote them. Fix: treat the trial component as an index, not a data store.
Re-using a TrialComponentDisplayName across unrelated jobs. It becomes hard to tell components apart when comparing later. Fix: name components so they're identifiable without opening each one.
Confusing this SDK-level Experiments API with the higher-level Python SDK's Run helper. They cover the same underlying resources but not always the same method names. Fix: pick one layer for a given project and check its current docs rather than mixing assumptions.

Alternatives

Alternative	Use When	Don't Use When
SageMaker Experiments (this page)	You want run metadata to live alongside SageMaker jobs with no extra infrastructure	You already run a dedicated experiment-tracking tool your team standardized on
Higher-level SageMaker Python SDK Run API	You want a more ergonomic wrapper and are comfortable tracking its version-specific API	You need the most stable, version-independent surface (use boto3 directly)
Third-party experiment tracking (self-hosted or SaaS)	Your team already has one in place across non-SageMaker workloads too	You want to avoid running or paying for extra infrastructure
Naming convention + S3 tags only	A handful of jobs, low iteration frequency	More than a few runs, or multiple people iterating on the same problem

FAQs

Do I need to call a separate logging API for basic training-job tracking?

No. Passing ExperimentConfig on CreateTrainingJob is enough - SageMaker creates and associates a trial component automatically, capturing the job's hyperparameters and any defined metrics.

What's the difference between an Experiment, a Trial, and a Trial Component?

An Experiment groups everything related to one problem. A Trial is one attempt at it. A Trial Component is one step of that attempt - most often a single training job, but it can be any logged step.

Can I log a custom metric that isn't produced by a training job?

Yes. Use CreateTrialComponent to create a component directly, then UpdateTrialComponent to add parameters or metrics, and AssociateTrialComponent to attach it to a trial.

How do I compare many trials at once?

Does SageMaker Experiments store my training data or model files?

No. It stores metadata - parameters, metrics, and references to where artifacts live. The actual data and model artifacts remain in S3 exactly where the training job wrote them.

Is there a simpler API than these boto3 calls?

SageMaker Training Basics - the training job this page attaches ExperimentConfig to.
Hyperparameter Tuning Jobs - a natural source of many trials to compare in one experiment.
Training Jobs: Built-in Algorithms vs Custom Containers - MetricDefinitions, shared with tuning and experiment tracking.
SageMaker Training via SDK Best Practices - reproducibility habits this page's tracking supports.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Experiment Tracking with SageMaker Experiments

Search Documentation

Experiment Tracking with SageMaker Experiments