Data Sources & Smart Parsing Ingestion

Multi-format ingestion (PDF, HTML, tables) without a custom parser.

A Knowledge Base only knows what its Data Source hands it after chunking and parsing. Get those two settings wrong and even a perfect vector store returns bad passages - chunks split mid-sentence, tables flattened into unreadable text, or PDFs losing their structure entirely. This page covers configuring both through the SDK, with a focus on the Smart Parsing option built for exactly that failure mode.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
ds = agent.create_data_source(
    knowledgeBaseId=kb_id,
    name="policy-pdfs",
    dataSourceConfiguration={
        "type": "S3",
        "s3Configuration": {"bucketArn": "arn:aws:s3:::my-policy-docs"},
    },
    vectorIngestionConfiguration={
        "chunkingConfiguration": {
            "chunkingStrategy": "FIXED_SIZE",
            "fixedSizeChunkingConfiguration": {"maxTokens": 300, "overlapPercentage": 20},
        },
        "parsingConfiguration": {
            "parsingStrategy": "BEDROCK_FOUNDATION_MODEL",
            "bedrockFoundationModelConfiguration": {
                "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0",
            },
        },
    },
)

// --- TypeScript (AWS SDK v3) ---
const ds = await agent.send(new CreateDataSourceCommand({
  knowledgeBaseId: kbId,
  name: "policy-pdfs",
  dataSourceConfiguration: {
    type: "S3",
    s3Configuration: { bucketArn: "arn:aws:s3:::my-policy-docs" },
  },
  vectorIngestionConfiguration: {
    chunkingConfiguration: {
      chunkingStrategy: "FIXED_SIZE",
      fixedSizeChunkingConfiguration: { maxTokens: 300, overlapPercentage: 20 },
    },
    parsingConfiguration: {
      parsingStrategy: "BEDROCK_FOUNDATION_MODEL",
      bedrockFoundationModelConfiguration: {
        modelArn: "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0",
      },
    },
  },
}));

When to reach for each setting:

• Fixed-size chunking - the safe default for most document sets; tune maxTokens and overlapPercentage, not much else to configure.
• Smart Parsing (BEDROCK_FOUNDATION_MODEL) - PDFs with tables/columns, HTML with structure that matters, or scanned-looking documents where plain-text extraction loses meaning.
• Skip Smart Parsing for clean plain-text or Markdown sources - the default parser already handles those well and costs less.

Working Example

Below, the same S3 Data Source is created twice: once with default parsing (fast, cheap, plain-text extraction) and once with Smart Parsing plus hierarchical chunking (slower, costlier, layout-aware) - so the trade-off is visible side by side.

# --- Python (boto3) ---
import boto3
 
agent = boto3.client("bedrock-agent", region_name="us-east-1")
 
# 1. Default parsing - plain text extraction, no vectorIngestionConfiguration needed.
default_ds = agent.create_data_source(
    knowledgeBaseId=kb_id,
    name="faqs-plain-text",
    dataSourceConfiguration={"type": "S3", "s3Configuration": {"bucketArn": "arn:aws:s3:::my-faqs"}},
)
 
# 2. Smart Parsing + hierarchical chunking - for PDFs with tables and nested sections.
smart_ds = agent.create_data_source(
    knowledgeBaseId=kb_id,
    name="contracts-smart-parsed",
    dataSourceConfiguration={"type": "S3", "s3Configuration": {"bucketArn": "arn:aws:s3:::my-contracts"}},
    vectorIngestionConfiguration={
        "chunkingConfiguration": {
            "chunkingStrategy": "HIERARCHICAL",
            "hierarchicalChunkingConfiguration": {
                "levelConfigurations": [{"maxTokens": 1500}, {"maxTokens": 300}],
                "overlapTokens": 60,
            },
        },
        "parsingConfiguration": {
            "parsingStrategy": "BEDROCK_FOUNDATION_MODEL",
            "bedrockFoundationModelConfiguration": {
                "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0",
                "parsingPrompt": {
                    "parsingPromptText": "Transcribe tables as Markdown tables and preserve section headings.",
                },
            },
        },
    },
)
print(default_ds["dataSource"]["dataSourceId"], smart_ds["dataSource"]["dataSourceId"])

// --- TypeScript (AWS SDK v3) ---
import { BedrockAgentClient, CreateDataSourceCommand } from "@aws-sdk/client-bedrock-agent";
 
const agent = new BedrockAgentClient({ region: "us-east-1" });
 
// 1. Default parsing - plain text extraction, no vectorIngestionConfiguration needed.
const defaultDs = await agent.send(new CreateDataSourceCommand({
  knowledgeBaseId: kbId,
  name: "faqs-plain-text",
  dataSourceConfiguration: { type: "S3", s3Configuration: { bucketArn: "arn:aws:s3:::my-faqs" } },
}));
 
// 2. Smart Parsing + hierarchical chunking - for PDFs with tables and nested sections.
const smartDs = await agent.send(new CreateDataSourceCommand({
  knowledgeBaseId: kbId,
  name: "contracts-smart-parsed",
  dataSourceConfiguration: { type: "S3", s3Configuration: { bucketArn: "arn:aws:s3:::my-contracts" } },
  vectorIngestionConfiguration: {
    chunkingConfiguration: {
      chunkingStrategy: "HIERARCHICAL",
      hierarchicalChunkingConfiguration: {
        levelConfigurations: [{ maxTokens: 1500 }, { maxTokens: 300 }],
        overlapTokens: 60,
      },
    },
    parsingConfiguration: {
      parsingStrategy: "BEDROCK_FOUNDATION_MODEL",
      bedrockFoundationModelConfiguration: {
        modelArn: "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0",
        parsingPrompt: {
          parsingPromptText: "Transcribe tables as Markdown tables and preserve section headings.",
        },
      },
    },
  },
}));
console.log(defaultDs.dataSource?.dataSourceId, smartDs.dataSource?.dataSourceId);

What this demonstrates:

• Omitting vectorIngestionConfiguration entirely gets you sane defaults - fixed-size chunking, plain-text parsing.
• parsingStrategy: "BEDROCK_FOUNDATION_MODEL" routes each document through a foundation model before chunking, so tables and layout survive as structured text.
• parsingPrompt.parsingPromptText steers how the model transcribes a document - here, forcing Markdown tables.
• Hierarchical chunking keeps a coarse parent chunk and a finer child chunk, useful when both broad context and precise passages matter.

Deep Dive

Chunking Strategies

chunkingStrategy accepts a few shapes, each with its own configuration block:

• FIXED_SIZE - split by an approximate token count (maxTokens) with a percentage overlapPercentage. Simple, predictable, the right default.
• HIERARCHICAL - produce nested chunks at multiple levelConfigurations sizes, useful for long structured documents (contracts, manuals) where you want both a broad and a narrow retrieval unit.
• SEMANTIC - split at natural topic boundaries rather than a fixed token count, using an embedding-based similarity check between sentences.
• NONE - one chunk per document; only sensible for already-short documents.

Chunk size is a trade-off, not a pure "bigger is better" knob. Small chunks retrieve precisely but lose surrounding context; large chunks preserve context but dilute the similarity signal and cost more per retrieved passage. overlapPercentage (or overlapTokens for hierarchical) softens hard boundary cuts by repeating a slice of text between adjacent chunks.

What Smart Parsing Actually Does

The default parser extracts text with a standard document-to-text conversion - fast and free, and fine for clean plain text, Markdown, or simple HTML. It struggles with PDFs that have multi-column layouts, embedded tables, or scanned-looking pages, because plain extraction reads text in the wrong order or flattens a table into an unreadable row of numbers.

parsingStrategy: "BEDROCK_FOUNDATION_MODEL" (the Smart Parsing / advanced parsing option) instead routes each document through a Bedrock foundation model during ingestion, asking it to transcribe the document's content, including tables and structure, as clean text. A parsingPrompt lets you steer that transcription - for example, requesting tables as Markdown, or asking the model to preserve section headings for better chunk boundaries.

This is strictly an ingestion-time setting per Data Source; it never applies at query time, and different Data Sources under the same Knowledge Base can use different parsing strategies.

The Cost and Latency Trade-off

Smart Parsing calls a foundation model once per document during ingestion, on top of the embedding call every chunk already needs. For a few hundred PDFs this is a rounding error; for a corpus of hundreds of thousands of scanned documents it is a real ingestion-time cost and duration to budget for. Reach for it selectively - enable it on the Data Sources whose documents actually have complex layout, and leave clean plain-text sources on the default parser.

Gotchas

• Applying Smart Parsing to already-clean text - you pay for a model call with no accuracy benefit. Fix: reserve it for PDFs, HTML with real structure, or documents with tables.
• Choosing chunk size without testing retrieval quality - too small loses context, too large dilutes similarity. Fix: start with FIXED_SIZE around 300-500 tokens and evaluate before tuning further.
• Changing chunking strategy after go-live and expecting old chunks to update - they do not; only new ingestion runs use the new configuration. Fix: re-run StartIngestionJob (or delete and recreate the Data Source) after a chunking change.
• Ignoring overlapPercentage - zero overlap means an answer that spans a chunk boundary can be split and never retrieved whole. Fix: set a modest overlap (10-20%) as a default.
• Assuming one parsing strategy covers every Data Source - a Knowledge Base can have Data Sources with different needs. Fix: configure vectorIngestionConfiguration per Data Source, not once globally.
• Forgetting the parsing model needs its own model access enabled - Smart Parsing calls a foundation model like any other Bedrock invocation. Fix: enable model access for the parsing model, same as any inference model.

Alternatives

FAQs

Related

• Bedrock Knowledge Bases Basics - creating the Data Source this page configures.
• Sync Jobs & Incremental Ingestion - keeping ingested content current after the first run.
• Managed RAG on Bedrock: What the Knowledge Base Owns - where ingestion fits in the full pipeline.
• Bedrock Knowledge Bases Best Practices - chunking and metadata recommendations.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+). Bedrock Knowledge Bases' Agentic Retriever is a fast-evolving 2026 capability - verify exact parameters against current docs.