Object, Scene & Label Detection

Most Rekognition integrations start here. DetectLabels is the pretrained operation that answers "what is in this image" with a ranked list of objects, scenes, and concepts - the backbone of auto-tagging, content search, and catalog enrichment.

This page goes past the basic call shown earlier in this section into the shape of the full response: label hierarchies, bounding boxes for countable objects, category filters, and the optional image-properties feature for dominant colors.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
rekognition = boto3.client("rekognition")
response = rekognition.detect_labels(
    Image={"S3Object": {"Bucket": "my-bucket", "Name": "catalog/sneaker.jpg"}},
    MaxLabels=10,
    MinConfidence=80,
)
for label in response["Labels"]:
    print(label["Name"], label["Confidence"])

// --- TypeScript (AWS SDK v3) ---
import { RekognitionClient, DetectLabelsCommand } from "@aws-sdk/client-rekognition";
 
const rekognition = new RekognitionClient({});
const response = await rekognition.send(new DetectLabelsCommand({
  Image: { S3Object: { Bucket: "my-bucket", Name: "catalog/sneaker.jpg" } },
  MaxLabels: 10,
  MinConfidence: 80,
}));
for (const label of response.Labels ?? []) console.log(label.Name, label.Confidence);

When to reach for this:

• Auto-tagging uploaded photos so they are searchable by content.
• Enriching a product or media catalog with descriptive metadata.
• Pre-filtering images before routing them to a more specific or expensive process.
• Counting or locating instances of a recurring object in a scene.

Working Example

Detect labels for a catalog photo, read the category hierarchy, pull bounding boxes for countable objects, and grab the image's dominant colors in the same call.

# --- Python (boto3) ---
import boto3
 
rekognition = boto3.client("rekognition")
 
response = rekognition.detect_labels(
    Image={"S3Object": {"Bucket": "my-bucket", "Name": "catalog/sneaker.jpg"}},
    MaxLabels=15,
    MinConfidence=75,
    Features=["GENERAL_LABELS", "IMAGE_PROPERTIES"],
)
 
for label in response["Labels"]:
    parents = [p["Name"] for p in label.get("Parents", [])]
    print(f"{label['Name']} ({label['Confidence']:.1f}) - under {parents}")
    for instance in label.get("Instances", []):
        print("  box:", instance["BoundingBox"])
 
colors = response.get("ImageProperties", {}).get("DominantColors", [])
for color in colors[:3]:
    print("dominant color:", color["SimplifiedColor"], color["PixelPercent"])

// --- TypeScript (AWS SDK v3) ---
import { RekognitionClient, DetectLabelsCommand } from "@aws-sdk/client-rekognition";
 
const rekognition = new RekognitionClient({});
 
const response = await rekognition.send(new DetectLabelsCommand({
  Image: { S3Object: { Bucket: "my-bucket", Name: "catalog/sneaker.jpg" } },
  MaxLabels: 15,
  MinConfidence: 75,
  Features: ["GENERAL_LABELS", "IMAGE_PROPERTIES"],
}));
 
for (const label of response.Labels ?? []) {
  const parents = (label.Parents ?? []).map((p) => p.Name);
  console.log(`${label.Name} (${label.Confidence?.toFixed(1)}) - under ${parents}`);
  for (const instance of label.Instances ?? []) {
    console.log("  box:", instance.BoundingBox);
  }
}
 
const colors = response.ImageProperties?.DominantColors ?? [];
for (const color of colors.slice(0, 3)) {
  console.log("dominant color:", color.SimplifiedColor, color.PixelPercent);
}

What this demonstrates:

• Parents places each label in a category hierarchy - "Sneaker" might sit under "Shoe" under "Footwear" under "Clothing".
• Instances appears only for countable objects and gives a BoundingBox (relative Left/Top/Width/Height) per occurrence.
• Features: ["IMAGE_PROPERTIES"] adds ImageProperties.DominantColors, useful for visual-similarity search alongside labels.
• Requesting Features explicitly is optional - omit it and you still get GENERAL_LABELS by default.

Deep Dive

The Label Hierarchy

Every label can carry a Parents list, forming a tree from specific to general. "Golden Retriever" might roll up through "Dog" and "Canine" to "Animal". This lets you tag at whatever granularity a use case needs - narrow for precise search, broad for filtering.

You can filter by name or by category directly in the request instead of post-processing the response, using Settings.GeneralLabels with LabelInclusionFilters, LabelExclusionFilters, LabelCategoryInclusionFilters, and LabelCategoryExclusionFilters. This moves filtering server-side, which reduces response size and simplifies client code.

response = rekognition.detect_labels(
    Image={"S3Object": {"Bucket": "my-bucket", "Name": "catalog/sneaker.jpg"}},
    Settings={
        "GeneralLabels": {
            "LabelCategoryInclusionFilters": ["Apparel and Accessories"],
        }
    },
)

Counting and Locating Objects

Only labels for countable, localizable things (people, vehicles, animals) include Instances. Abstract labels like "Outdoors" or "Daytime" describe the whole image and never carry bounding boxes. If your use case is "how many people are in this frame", read len(label["Instances"]) for the Person label rather than trying to count a scene label.

Image Properties

Features: ["IMAGE_PROPERTIES"] runs a separate analysis alongside labels and returns overall image quality (Quality.Brightness, Quality.Sharpness, Quality.Contrast) plus DominantColors, each with an RGB value, a SimplifiedColor name, and the percentage of pixels it covers. This is useful for building "find visually similar" search without a separate color-analysis step.

Choosing MinConfidence and MaxLabels

MinConfidence (default 55) filters weak matches before they reach you; raise it for a cleaner but shorter list, lower it if you would rather see borderline candidates and filter client-side. MaxLabels (default 1,000) caps the response size - most catalog use cases set it in the 10-20 range since relevance drops fast past the top labels.

Gotchas

• Treating labels as ground truth. Confidence is a model estimate, not a certainty. Fix: validate MinConfidence against a labeled sample from your own domain before trusting a threshold in production.
• Expecting Instances on every label. Only countable object labels carry bounding boxes; scene and concept labels never do. Fix: check label.get("Instances") for emptiness rather than assuming it exists.
• Sending images over 5 MB as bytes. The Image.Bytes path is capped at 5 MB. Fix: upload to S3 first and use S3Object for larger images, which has a higher limit.
• Ignoring the category hierarchy. Filtering only on top-level label names misses synonyms and sibling labels under the same parent. Fix: use LabelCategoryInclusionFilters or walk Parents when you need category-level matching.
• Assuming Features defaults include IMAGE_PROPERTIES. Only GENERAL_LABELS is returned unless you request IMAGE_PROPERTIES explicitly. Fix: pass Features: ["GENERAL_LABELS", "IMAGE_PROPERTIES"] when you need both.
• Re-running DetectLabels on unchanged images. Every call is billed regardless of whether the image was analyzed before. Fix: cache results keyed by object version or content hash.

Alternatives

FAQs

Related

• Rekognition Basics - the first DetectLabels call and response shape.
• Rekognition's Pretrained Vision Model Surface - where DetectLabels sits relative to Custom Labels.
• Video Analysis with Asynchronous Operations - label detection across a full video.
• Custom Labels for Domain-Specific Detection - detecting what general labels miss.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).