Q Business: Data Sources & Indexing

Q: Do I need to run a sync before I can ask questions?

Yes. ChatSync only answers from content that has completed at least one successful StartDataSourceSyncJob run against the index.

Q: How do I know a sync actually worked?

Poll ListDataSourceSyncJobs, wait for the latest entry's status to reach SUCCEEDED, and check its metrics for a sane number of documents added or modified.

Q: What's the most common reason a sync fails?

The service role passed to CreateDataSource lacking read permission on the actual source (bucket, site, or prefix). Check the job's error field first.

Q: Should I schedule syncs myself?

Yes, typically. Put StartDataSourceSyncJob behind your own schedule (for example EventBridge plus Lambda) matching how often the underlying content changes.

Connecting enterprise data sources for Q&A via SDK.

An Amazon Q Business application can only answer from content it has actually indexed. That happens in two steps: you register a data source with CreateDataSource, telling Q Business what to connect to and how, and then you run a sync job with StartDataSourceSyncJob, which does the real work of crawling, extracting, and embedding that content into the index. This page covers both steps, the connector configuration shape, and the sync lifecycle you will monitor in production.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
resp = qbiz.create_data_source(
    applicationId=application_id, indexId=index_id,
    displayName="docs-s3",
    configuration={"type": "S3", "syncMode": "FULL_CRAWL",
                   "connectionConfiguration": {"repositoryEndpointMetadata": {"BucketName": "my-bucket"}}},
    roleArn="arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
)
qbiz.start_data_source_sync_job(applicationId=application_id, indexId=index_id, dataSourceId=resp["dataSourceId"])

// --- TypeScript (AWS SDK v3) ---
const resp = await qbiz.send(new CreateDataSourceCommand({
  applicationId, indexId, displayName: "docs-s3",
  configuration: { type: "S3", syncMode: "FULL_CRAWL",
    connectionConfiguration: { repositoryEndpointMetadata: { BucketName: "my-bucket" } } },
  roleArn: "arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
}));
await qbiz.send(new StartDataSourceSyncJobCommand({ applicationId, indexId, dataSourceId: resp.dataSourceId }));

When to reach for this:

Bringing an existing knowledge base (S3 documents, SharePoint sites, Confluence spaces, or public web pages) into a Q Business index.
Keeping an index current as source content changes, via scheduled or repeated syncs.
Standing up several data sources feeding the same index, so one assistant spans multiple systems.
Skip it when the content already lives in a format Q Business's ChatSync can be given directly, without needing a persistent index.

Working Example

# --- Python (boto3) ---
import time
import boto3
 
qbiz = boto3.client("qbusiness", region_name="us-east-1")
 
create_resp = qbiz.create_data_source(
    applicationId=application_id,
    indexId=index_id,
    displayName="engineering-docs",
    configuration={
        "type": "S3",
        "syncMode": "FULL_CRAWL",
        "connectionConfiguration": {
            "repositoryEndpointMetadata": {"BucketName": "eng-docs-bucket", "Prefix": "public/"}
        },
    },
    roleArn="arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
)
data_source_id = create_resp["dataSourceId"]
 
qbiz.start_data_source_sync_job(
    applicationId=application_id, indexId=index_id, dataSourceId=data_source_id,
)
 
while True:
    history = qbiz.list_data_source_sync_jobs(
        applicationId=application_id, indexId=index_id, dataSourceId=data_source_id,
    )["history"]
    job = history[0] if history else None
    status = job["status"] if job else "UNKNOWN"
    print("sync status:", status)
    if status == "FAILED":
        print("error:", job.get("error"))
        break
    if status == "SUCCEEDED":
        print("documents added/modified/deleted:", job.get("metrics"))
        break
    time.sleep(15)

// --- TypeScript (AWS SDK v3) ---
import {
  QBusinessClient,
  CreateDataSourceCommand,
  StartDataSourceSyncJobCommand,
  ListDataSourceSyncJobsCommand,
} from "@aws-sdk/client-qbusiness";
 
const qbiz = new QBusinessClient({ region: "us-east-1" });
 
const createResp = await qbiz.send(new CreateDataSourceCommand({
  applicationId,
  indexId,
  displayName: "engineering-docs",
  configuration: {
    type: "S3",
    syncMode: "FULL_CRAWL",
    connectionConfiguration: {
      repositoryEndpointMetadata: { BucketName: "eng-docs-bucket", Prefix: "public/" },
    },
  },
  roleArn: "arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
}));
const dataSourceId = createResp.dataSourceId!;
 
await qbiz.send(new StartDataSourceSyncJobCommand({ applicationId, indexId, dataSourceId }));
 
let status = "UNKNOWN";
while (status !== "SUCCEEDED" && status !== "FAILED") {
  const history = (await qbiz.send(new ListDataSourceSyncJobsCommand({ applicationId, indexId, dataSourceId }))).history ?? [];
  const job = history[0];
  status = job?.status ?? "UNKNOWN";
  console.log("sync status:", status);
  if (status === "FAILED") console.log("error:", job?.error);
  if (status === "SUCCEEDED") console.log("metrics:", job?.metrics);
  if (status !== "SUCCEEDED" && status !== "FAILED") await new Promise((r) => setTimeout(r, 15_000));
}

What this demonstrates:

configuration carries a connector type (S3 here) plus a connector-specific connectionConfiguration block.
CreateDataSource only registers the connection - no content moves until StartDataSourceSyncJob runs.
ListDataSourceSyncJobs's most recent entry carries status, and on completion, error or metrics you should read.
The same poll-until-terminal-status pattern used for application and index creation applies to sync jobs too.

Deep Dive

Connector Types

Q Business ships prebuilt connectors for common enterprise systems: Amazon S3, SharePoint, Confluence, a generic web crawler, and others, each configured through its own shape under configuration. A custom connector option also exists for sources without a built-in integration. Each connector type expects different fields inside connectionConfiguration and often its own set of required IAM permissions on the service role - always check the current connector reference for the one you are using rather than assuming S3's shape generalizes.

Sync Modes and Scheduling

syncMode (or an equivalent field depending on connector) controls whether a run reprocesses everything (FULL_CRAWL) or only picks up what changed since the previous successful sync. A full crawl costs more time and, depending on content volume, more index capacity to process, so most production setups run a full crawl once and incremental syncs afterward. You can trigger a sync on demand via StartDataSourceSyncJob, or put that call behind your own schedule (an EventBridge rule invoking a Lambda, for example) if the source changes on a predictable cadence.

Reading Sync Results

Every sync job produces a history entry with a terminal status of SUCCEEDED, FAILED, or occasionally partial success depending on the connector. On success, metrics typically reports documents added, modified, and deleted - a sudden drop to zero added documents on a source you know changed is worth investigating before trusting the index. On failure, the job's error field is the fastest path to a root cause, most often an IAM permission gap on the service role or a malformed connector configuration.

Multiple Data Sources, One Index

An index is not limited to one data source. You can attach an S3 connector for internal documents and a Confluence connector for wiki pages to the same index, and ChatSync answers from the combined corpus with source attributions telling you which connector a given answer came from. Each data source syncs independently, so one connector's failure does not block another's.

Gotchas

Assuming CreateDataSource crawls content - it only registers the connection. Fix: always follow with StartDataSourceSyncJob and confirm it reaches SUCCEEDED.
Service role missing read permission on the source - syncs fail immediately. Fix: grant the role read access to the specific S3 bucket/prefix, SharePoint site, or equivalent, before creating the data source.
Expecting instant answers after a sync starts - ChatSync still reflects the old index until the job reaches SUCCEEDED. Fix: poll ListDataSourceSyncJobs and wait for a terminal status before testing.
Running a full crawl on every schedule - wastes time and capacity on unchanged content. Fix: use an incremental sync mode for routine refreshes once the first full crawl succeeds.
Ignoring metrics/error on completed jobs - a SUCCEEDED sync with zero documents processed can still mean something is wrong. Fix: check the counts, not just the status string.
Hardcoding connector configuration shapes from memory - connector-specific fields change across versions. Fix: verify the exact configuration shape for your connector type against the current qbusiness API reference before deploying.

Alternatives

Approach	Use When	Don't Use When
Q Business data source + sync (this page)	You want a persistent, searchable index over evolving enterprise content	The content is small and static enough to pass directly per request
Passing context directly in a Bedrock Converse call	The relevant content fits in one prompt and rarely changes	You need indexed search across a large, growing corpus with citations
A custom RAG pipeline (your own embeddings + vector store)	You need control over chunking, embeddings, or a vector store Q Business doesn't support	You want a managed connector/sync/chat experience without building retrieval yourself
Amazon Kendra as the retrieval layer	You already have a Kendra index and want to keep it separate from Q Business	You want Q Business's integrated chat and plugin experience on top

FAQs

Do I need to run a sync before I can ask questions?

Yes. ChatSync only answers from content that has completed at least one successful StartDataSourceSyncJob run against the index.

What connectors does Q Business support?

Prebuilt connectors cover common enterprise sources including Amazon S3, SharePoint, Confluence, and a web crawler, plus custom connector options for other systems - check current AWS docs for the full, evolving list.

How do I know a sync actually worked?

Poll ListDataSourceSyncJobs, wait for the latest entry's status to reach SUCCEEDED, and check its metrics for a sane number of documents added or modified.

What's the difference between a full crawl and an incremental sync?

A full crawl reprocesses the entire source; an incremental sync only picks up changes since the last successful run. Use full for the first sync, incremental afterward.

Can one index have multiple data sources?

Yes. An index can combine several connectors - for example S3 and Confluence - and ChatSync answers from the combined content, citing which source each attribution came from.

What's the most common reason a sync fails?

The service role passed to CreateDataSource lacking read permission on the actual source (bucket, site, or prefix). Check the job's error field first.

Should I schedule syncs myself?

Yes, typically. Put StartDataSourceSyncJob behind your own schedule (for example EventBridge plus Lambda) matching how often the underlying content changes.

Amazon Q Basics - the application and index this data source attaches to.
Q Business Chat & Plugin Integrations - querying the index you just built.
Access Control & Guardrails for Q Business - scoping who can see answers from which sources.
Amazon Q via SDK Best Practices - data-source and sync hygiene at a glance.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
resp = qbiz.create_data_source(
    applicationId=application_id, indexId=index_id,
    displayName="docs-s3",
    configuration={"type": "S3", "syncMode": "FULL_CRAWL",
                   "connectionConfiguration": {"repositoryEndpointMetadata": {"BucketName": "my-bucket"}}},
    roleArn="arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
)
qbiz.start_data_source_sync_job(applicationId=application_id, indexId=index_id, dataSourceId=resp["dataSourceId"])

// --- TypeScript (AWS SDK v3) ---
const resp = await qbiz.send(new CreateDataSourceCommand({
  applicationId, indexId, displayName: "docs-s3",
  configuration: { type: "S3", syncMode: "FULL_CRAWL",
    connectionConfiguration: { repositoryEndpointMetadata: { BucketName: "my-bucket" } } },
  roleArn: "arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
}));
await qbiz.send(new StartDataSourceSyncJobCommand({ applicationId, indexId, dataSourceId: resp.dataSourceId }));

When to reach for this:

Bringing an existing knowledge base (S3 documents, SharePoint sites, Confluence spaces, or public web pages) into a Q Business index.
Keeping an index current as source content changes, via scheduled or repeated syncs.
Standing up several data sources feeding the same index, so one assistant spans multiple systems.
Skip it when the content already lives in a format Q Business's ChatSync can be given directly, without needing a persistent index.

Working Example

# --- Python (boto3) ---
import time
import boto3
 
qbiz = boto3.client("qbusiness", region_name="us-east-1")
 
create_resp = qbiz.create_data_source(
    applicationId=application_id,
    indexId=index_id,
    displayName="engineering-docs",
    configuration={
        "type": "S3",
        "syncMode": "FULL_CRAWL",
        "connectionConfiguration": {
            "repositoryEndpointMetadata": {"BucketName": "eng-docs-bucket", "Prefix": "public/"}
        },
    },
    roleArn="arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
)
data_source_id = create_resp["dataSourceId"]
 
qbiz.start_data_source_sync_job(
    applicationId=application_id, indexId=index_id, dataSourceId=data_source_id,
)
 
while True:
    history = qbiz.list_data_source_sync_jobs(
        applicationId=application_id, indexId=index_id, dataSourceId=data_source_id,
    )["history"]
    job = history[0] if history else None
    status = job["status"] if job else "UNKNOWN"
    print("sync status:", status)
    if status == "FAILED":
        print("error:", job.get("error"))
        break
    if status == "SUCCEEDED":
        print("documents added/modified/deleted:", job.get("metrics"))
        break
    time.sleep(15)

// --- TypeScript (AWS SDK v3) ---
import {
  QBusinessClient,
  CreateDataSourceCommand,
  StartDataSourceSyncJobCommand,
  ListDataSourceSyncJobsCommand,
} from "@aws-sdk/client-qbusiness";
 
const qbiz = new QBusinessClient({ region: "us-east-1" });
 
const createResp = await qbiz.send(new CreateDataSourceCommand({
  applicationId,
  indexId,
  displayName: "engineering-docs",
  configuration: {
    type: "S3",
    syncMode: "FULL_CRAWL",
    connectionConfiguration: {
      repositoryEndpointMetadata: { BucketName: "eng-docs-bucket", Prefix: "public/" },
    },
  },
  roleArn: "arn:aws:iam::123456789012:role/QBusinessS3ReaderRole",
}));
const dataSourceId = createResp.dataSourceId!;
 
await qbiz.send(new StartDataSourceSyncJobCommand({ applicationId, indexId, dataSourceId }));
 
let status = "UNKNOWN";
while (status !== "SUCCEEDED" && status !== "FAILED") {
  const history = (await qbiz.send(new ListDataSourceSyncJobsCommand({ applicationId, indexId, dataSourceId }))).history ?? [];
  const job = history[0];
  status = job?.status ?? "UNKNOWN";
  console.log("sync status:", status);
  if (status === "FAILED") console.log("error:", job?.error);
  if (status === "SUCCEEDED") console.log("metrics:", job?.metrics);
  if (status !== "SUCCEEDED" && status !== "FAILED") await new Promise((r) => setTimeout(r, 15_000));
}

What this demonstrates:

configuration carries a connector type (S3 here) plus a connector-specific connectionConfiguration block.
CreateDataSource only registers the connection - no content moves until StartDataSourceSyncJob runs.
ListDataSourceSyncJobs's most recent entry carries status, and on completion, error or metrics you should read.
The same poll-until-terminal-status pattern used for application and index creation applies to sync jobs too.

Deep Dive

Connector Types

Sync Modes and Scheduling

Reading Sync Results

Multiple Data Sources, One Index

Gotchas

Assuming CreateDataSource crawls content - it only registers the connection. Fix: always follow with StartDataSourceSyncJob and confirm it reaches SUCCEEDED.
Service role missing read permission on the source - syncs fail immediately. Fix: grant the role read access to the specific S3 bucket/prefix, SharePoint site, or equivalent, before creating the data source.
Expecting instant answers after a sync starts - ChatSync still reflects the old index until the job reaches SUCCEEDED. Fix: poll ListDataSourceSyncJobs and wait for a terminal status before testing.
Running a full crawl on every schedule - wastes time and capacity on unchanged content. Fix: use an incremental sync mode for routine refreshes once the first full crawl succeeds.
Ignoring metrics/error on completed jobs - a SUCCEEDED sync with zero documents processed can still mean something is wrong. Fix: check the counts, not just the status string.
Hardcoding connector configuration shapes from memory - connector-specific fields change across versions. Fix: verify the exact configuration shape for your connector type against the current qbusiness API reference before deploying.

Alternatives

Approach	Use When	Don't Use When
Q Business data source + sync (this page)	You want a persistent, searchable index over evolving enterprise content	The content is small and static enough to pass directly per request
Passing context directly in a Bedrock Converse call	The relevant content fits in one prompt and rarely changes	You need indexed search across a large, growing corpus with citations
A custom RAG pipeline (your own embeddings + vector store)	You need control over chunking, embeddings, or a vector store Q Business doesn't support	You want a managed connector/sync/chat experience without building retrieval yourself
Amazon Kendra as the retrieval layer	You already have a Kendra index and want to keep it separate from Q Business	You want Q Business's integrated chat and plugin experience on top

FAQs

Do I need to run a sync before I can ask questions?

Yes. ChatSync only answers from content that has completed at least one successful StartDataSourceSyncJob run against the index.

What connectors does Q Business support?

How do I know a sync actually worked?

Poll ListDataSourceSyncJobs, wait for the latest entry's status to reach SUCCEEDED, and check its metrics for a sane number of documents added or modified.

What's the difference between a full crawl and an incremental sync?

A full crawl reprocesses the entire source; an incremental sync only picks up changes since the last successful run. Use full for the first sync, incremental afterward.

Can one index have multiple data sources?

Yes. An index can combine several connectors - for example S3 and Confluence - and ChatSync answers from the combined content, citing which source each attribution came from.

What's the most common reason a sync fails?

The service role passed to CreateDataSource lacking read permission on the actual source (bucket, site, or prefix). Check the job's error field first.

Should I schedule syncs myself?

Yes, typically. Put StartDataSourceSyncJob behind your own schedule (for example EventBridge plus Lambda) matching how often the underlying content changes.

Amazon Q Basics - the application and index this data source attaches to.
Q Business Chat & Plugin Integrations - querying the index you just built.
Access Control & Guardrails for Q Business - scoping who can see answers from which sources.
Amazon Q via SDK Best Practices - data-source and sync hygiene at a glance.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Q Business: Data Sources & Indexing

Search Documentation

Q Business: Data Sources & Indexing