S3 Vectors via SDK Best Practices

Q: How do I write vectors efficiently?

Batch many per put_vectors call rather than one per request, use stable deterministic keys so re-runs upsert, and retry with backoff. Size the batch to the service maximum, which you should verify against current docs.

Q: Why are metadata and distance missing from my results?

They are opt-in. Set returnMetadata and returnDistance to true, and include s3vectors:GetVectors in your IAM policy, which metadata access requires alongside QueryVectors.

Q: How do I keep costs down?

Batch writes to cut request count, keep topK and returned metadata minimal, delete stale vectors, and estimate Bedrock embedding cost separately since batching does not reduce it. Use S3 Vectors for the latency-tolerant workloads it is priced for.

This is the checklist to run before and after taking an S3 Vectors workload to production from the SDK.

Each item is a rule stated positively, with a one-line rationale. Work top to bottom - the groups run from how you design the index, through writing and querying, to cost and security. Most rules apply whether you call S3 Vectors from boto3 or the AWS SDK v3. Because this is a 2026 API, several rules end with a reminder to verify specifics against current docs.

How to Use This List

Treat each box as an audit question against your own project.
The groups run from index design outward to cost and security.
Verify anything marked as a 2026-API specific (batch limits, filter grammar, operation names) against live docs before you rely on it.
Revisit the list whenever the corpus grows, the embedding model changes, or costs surprise you.

A - Index Design

Match dimension to your embedding model. Titan Text Embeddings v2 is 1024; the index dimension must equal it exactly or writes fail.
Pick the distance metric your model recommends. cosine for most text embeddings, euclidean for some others - and it is fixed at creation.
Treat dimension and metric as immutable. Changing either means a new index and a full reload, so decide before loading millions of vectors.
Declare non-filterable metadata keys. Mark bulky fields like raw text as nonFilterableMetadataKeys so they store but do not bloat the filterable set.

Create the index deliberately - it is the one decision you cannot walk back cheaply.

# --- Python (boto3) ---
import boto3
 
sv = boto3.client("s3vectors")
sv.create_index(
    vectorBucketName="my-vectors", indexName="docs",
    dataType="float32", dimension=1024, distanceMetric="cosine",
    metadataConfiguration={"nonFilterableMetadataKeys": ["raw_text"]},
)
# Verify operation/parameter names against current boto3 docs (2026 API surface).

// --- TypeScript (AWS SDK v3) ---
import { S3VectorsClient, CreateIndexCommand } from "@aws-sdk/client-s3vectors";
 
const sv = new S3VectorsClient({});
await sv.send(new CreateIndexCommand({
  vectorBucketName: "my-vectors", indexName: "docs",
  dataType: "float32", dimension: 1024, distanceMetric: "cosine",
  metadataConfiguration: { nonFilterableMetadataKeys: ["raw_text"] },
}));
// Verify operation/parameter names against current @aws-sdk/client-s3vectors docs (2026 API surface).

B - Writing Vectors Efficiently

Batch many vectors per put_vectors call. A list write beats one-per-request on both cost and time - size the batch to the service maximum.
Use stable, deterministic keys. Derive the key from the source id or a content hash so re-runs upsert instead of duplicating.
Validate embeddings before writing. Reject NaN/Infinity and all-zero vectors (invalid under cosine) so a bad batch does not fail late.
Make ingestion idempotent and resumable. Combine stable keys with checkpointing so a crashed load safely re-runs.

C - Querying Well

Keep topK small for RAG. 3 to 10 neighbors give enough context without weak matches; retrieve more only if you re-rank.
Opt into returnMetadata and returnDistance only when needed. They default off; requesting them costs more and needs s3vectors:GetVectors.
Filter on metadata during the query. A filter prunes candidates in search, returning the true top-k of the matching subset rather than filtering after.
Embed queries with the same model and dimension as the stored vectors. Mixing models or lengths makes distances meaningless or fails outright.

Ask only for what you use - metadata and distance are opt-in.

# --- Python (boto3) ---
res = sv.query_vectors(
    vectorBucketName="my-vectors", indexName="docs",
    topK=5, queryVector={"float32": q},
    filter={"topic": "iam"},          # verify filter grammar at build time (2026 API)
    returnMetadata=True, returnDistance=True,
)

// --- TypeScript (AWS SDK v3) ---
const res = await sv.send(new QueryVectorsCommand({
  vectorBucketName: "my-vectors", indexName: "docs",
  topK: 5, queryVector: { float32: q },
  filter: { topic: "iam" },           // verify filter grammar at build time (2026 API)
  returnMetadata: true, returnDistance: true,
}));

D - Cost Control

Batch to cut request count. Fewer, larger put_vectors and paged reads lower the per-request bill that dominates high-volume work.
Estimate embedding cost separately. Bedrock charges per token; batching writes does not change it, so size big backfills before running them.
Right-size topK and metadata returns. Smaller results and opt-in metadata reduce per-query work and cost.
Delete vectors you no longer need. Prune stale keys so storage stops billing for dead data.
Choose S3 Vectors for the workloads it fits. Its economics win on large, latency-tolerant corpora - do not force it onto low-latency serving where a warm index is cheaper overall.

E - Reliability

Reuse one client across calls. A module-scoped s3vectors client shares the connection pool and resolved credentials.
Retry writes and embeds with backoff. Transient throttling should back off, not abort a multi-million-vector load.
Inspect batch responses for per-vector errors. A batch can partially fail; retry only the failed keys, which idempotent keys make safe.
Pin one embedding model per index. Consistent dimension and semantics keep results comparable and writes valid.

Tune retries once on the client rather than wrapping every call.

# --- Python (boto3) ---
import boto3
from botocore.config import Config
 
sv = boto3.client("s3vectors", config=Config(retries={"max_attempts": 10, "mode": "adaptive"}))

// --- TypeScript (AWS SDK v3) ---
import { S3VectorsClient } from "@aws-sdk/client-s3vectors";
 
const sv = new S3VectorsClient({ maxAttempts: 10, retryMode: "adaptive" });

F - Security

Encrypt sensitive data with SSE-KMS. Pass encryptionConfiguration with sseType: aws:kms for a customer-managed key instead of default SSE-S3.
Scope IAM to specific bucket and index ARNs. Grant s3vectors actions on your exact resources, not s3vectors:* on *.
Grant GetVectors only where metadata is needed. Metadata filtering and returnMetadata require s3vectors:GetVectors on top of QueryVectors - do not grant it broadly.
Keep secrets out of metadata. Metadata is returned in queries; store only what callers may see, and put sensitive text behind non-filterable keys or elsewhere.
Isolate tenants by index or metadata filter. Separate indexes or an enforced tenant filter prevent one tenant's query from reaching another's vectors.

FAQs

What is the single most important design decision?

The index dimension and distance metric. Both are immutable, must match your embedding model, and changing either forces a new index and a full reload of every vector.

How do I write vectors efficiently?

Batch many per put_vectors call rather than one per request, use stable deterministic keys so re-runs upsert, and retry with backoff. Size the batch to the service maximum, which you should verify against current docs.

Why are metadata and distance missing from my results?

They are opt-in. Set returnMetadata and returnDistance to true, and include s3vectors:GetVectors in your IAM policy, which metadata access requires alongside QueryVectors.

How do I keep costs down?

Batch writes to cut request count, keep topK and returned metadata minimal, delete stale vectors, and estimate Bedrock embedding cost separately since batching does not reduce it. Use S3 Vectors for the latency-tolerant workloads it is priced for.

How should I secure a vector index?

Encrypt sensitive data with SSE-KMS, scope IAM to specific bucket/index ARNs, grant GetVectors narrowly, keep secrets out of returnable metadata, and isolate tenants by index or an enforced metadata filter.

How do I avoid duplicate vectors?

Use a deterministic key derived from the source id or a content hash. Writing the same key overwrites in place, so reprocessing a corpus upserts rather than creating duplicates.

Do these rules change because the API is new?

The habits hold, but specifics move. Verify operation names, the put_vectors batch maximum, and the metadata filter grammar against current docs before you depend on them - this is a 2026 API surface.

When should I not use S3 Vectors at all?

When you need consistently low-single-digit-millisecond queries, very high QPS, or hybrid keyword-plus-vector search. Those belong on OpenSearch, Pinecone, or pgvector - see the comparison page.

S3 Vectors Basics - the create/write/query building blocks these rules refine.
Batch-Loading Embeddings at Scale - the batching and idempotency rules in group B applied.
Vector Indexes & Similarity Search Queries - the query patterns in group C.
S3 Vectors vs Self-Hosted Vector Databases - when S3 Vectors is the right store at all.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+). Amazon S3 Vectors is a fast-moving 2026 API - verify exact operation and parameter names against current docs.

S3 Vectors via SDK Best Practices

This is the checklist to run before and after taking an S3 Vectors workload to production from the SDK.

How to Use This List

Treat each box as an audit question against your own project.
The groups run from index design outward to cost and security.
Verify anything marked as a 2026-API specific (batch limits, filter grammar, operation names) against live docs before you rely on it.
Revisit the list whenever the corpus grows, the embedding model changes, or costs surprise you.

A - Index Design

Match dimension to your embedding model. Titan Text Embeddings v2 is 1024; the index dimension must equal it exactly or writes fail.
Pick the distance metric your model recommends. cosine for most text embeddings, euclidean for some others - and it is fixed at creation.
Treat dimension and metric as immutable. Changing either means a new index and a full reload, so decide before loading millions of vectors.
Declare non-filterable metadata keys. Mark bulky fields like raw text as nonFilterableMetadataKeys so they store but do not bloat the filterable set.

Create the index deliberately - it is the one decision you cannot walk back cheaply.

# --- Python (boto3) ---
import boto3
 
sv = boto3.client("s3vectors")
sv.create_index(
    vectorBucketName="my-vectors", indexName="docs",
    dataType="float32", dimension=1024, distanceMetric="cosine",
    metadataConfiguration={"nonFilterableMetadataKeys": ["raw_text"]},
)
# Verify operation/parameter names against current boto3 docs (2026 API surface).

// --- TypeScript (AWS SDK v3) ---
import { S3VectorsClient, CreateIndexCommand } from "@aws-sdk/client-s3vectors";
 
const sv = new S3VectorsClient({});
await sv.send(new CreateIndexCommand({
  vectorBucketName: "my-vectors", indexName: "docs",
  dataType: "float32", dimension: 1024, distanceMetric: "cosine",
  metadataConfiguration: { nonFilterableMetadataKeys: ["raw_text"] },
}));
// Verify operation/parameter names against current @aws-sdk/client-s3vectors docs (2026 API surface).

B - Writing Vectors Efficiently

Batch many vectors per put_vectors call. A list write beats one-per-request on both cost and time - size the batch to the service maximum.
Use stable, deterministic keys. Derive the key from the source id or a content hash so re-runs upsert instead of duplicating.
Validate embeddings before writing. Reject NaN/Infinity and all-zero vectors (invalid under cosine) so a bad batch does not fail late.
Make ingestion idempotent and resumable. Combine stable keys with checkpointing so a crashed load safely re-runs.

C - Querying Well

Keep topK small for RAG. 3 to 10 neighbors give enough context without weak matches; retrieve more only if you re-rank.
Opt into returnMetadata and returnDistance only when needed. They default off; requesting them costs more and needs s3vectors:GetVectors.
Filter on metadata during the query. A filter prunes candidates in search, returning the true top-k of the matching subset rather than filtering after.
Embed queries with the same model and dimension as the stored vectors. Mixing models or lengths makes distances meaningless or fails outright.

Ask only for what you use - metadata and distance are opt-in.

# --- Python (boto3) ---
res = sv.query_vectors(
    vectorBucketName="my-vectors", indexName="docs",
    topK=5, queryVector={"float32": q},
    filter={"topic": "iam"},          # verify filter grammar at build time (2026 API)
    returnMetadata=True, returnDistance=True,
)

// --- TypeScript (AWS SDK v3) ---
const res = await sv.send(new QueryVectorsCommand({
  vectorBucketName: "my-vectors", indexName: "docs",
  topK: 5, queryVector: { float32: q },
  filter: { topic: "iam" },           // verify filter grammar at build time (2026 API)
  returnMetadata: true, returnDistance: true,
}));

D - Cost Control

Batch to cut request count. Fewer, larger put_vectors and paged reads lower the per-request bill that dominates high-volume work.
Estimate embedding cost separately. Bedrock charges per token; batching writes does not change it, so size big backfills before running them.
Right-size topK and metadata returns. Smaller results and opt-in metadata reduce per-query work and cost.
Delete vectors you no longer need. Prune stale keys so storage stops billing for dead data.
Choose S3 Vectors for the workloads it fits. Its economics win on large, latency-tolerant corpora - do not force it onto low-latency serving where a warm index is cheaper overall.

E - Reliability

Reuse one client across calls. A module-scoped s3vectors client shares the connection pool and resolved credentials.
Retry writes and embeds with backoff. Transient throttling should back off, not abort a multi-million-vector load.
Inspect batch responses for per-vector errors. A batch can partially fail; retry only the failed keys, which idempotent keys make safe.
Pin one embedding model per index. Consistent dimension and semantics keep results comparable and writes valid.

Tune retries once on the client rather than wrapping every call.

# --- Python (boto3) ---
import boto3
from botocore.config import Config
 
sv = boto3.client("s3vectors", config=Config(retries={"max_attempts": 10, "mode": "adaptive"}))

// --- TypeScript (AWS SDK v3) ---
import { S3VectorsClient } from "@aws-sdk/client-s3vectors";
 
const sv = new S3VectorsClient({ maxAttempts: 10, retryMode: "adaptive" });

F - Security

Encrypt sensitive data with SSE-KMS. Pass encryptionConfiguration with sseType: aws:kms for a customer-managed key instead of default SSE-S3.
Scope IAM to specific bucket and index ARNs. Grant s3vectors actions on your exact resources, not s3vectors:* on *.
Grant GetVectors only where metadata is needed. Metadata filtering and returnMetadata require s3vectors:GetVectors on top of QueryVectors - do not grant it broadly.
Keep secrets out of metadata. Metadata is returned in queries; store only what callers may see, and put sensitive text behind non-filterable keys or elsewhere.
Isolate tenants by index or metadata filter. Separate indexes or an enforced tenant filter prevent one tenant's query from reaching another's vectors.

FAQs

What is the single most important design decision?

The index dimension and distance metric. Both are immutable, must match your embedding model, and changing either forces a new index and a full reload of every vector.

How do I write vectors efficiently?

Why are metadata and distance missing from my results?

They are opt-in. Set returnMetadata and returnDistance to true, and include s3vectors:GetVectors in your IAM policy, which metadata access requires alongside QueryVectors.

How do I keep costs down?

How should I secure a vector index?

How do I avoid duplicate vectors?

Use a deterministic key derived from the source id or a content hash. Writing the same key overwrites in place, so reprocessing a corpus upserts rather than creating duplicates.

Do these rules change because the API is new?

When should I not use S3 Vectors at all?

When you need consistently low-single-digit-millisecond queries, very high QPS, or hybrid keyword-plus-vector search. Those belong on OpenSearch, Pinecone, or pgvector - see the comparison page.

S3 Vectors Basics - the create/write/query building blocks these rules refine.
Batch-Loading Embeddings at Scale - the batching and idempotency rules in group B applied.
Vector Indexes & Similarity Search Queries - the query patterns in group C.
S3 Vectors vs Self-Hosted Vector Databases - when S3 Vectors is the right store at all.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+). Amazon S3 Vectors is a fast-moving 2026 API - verify exact operation and parameter names against current docs.

S3 Vectors via SDK Best Practices

Buscar na documentação

S3 Vectors via SDK Best Practices