Multipart Upload & the Transfer Manager

Q: How do I tune throughput?

Raise concurrency (max_concurrency in boto3, queueSize in v3) and part size for big files on fast links. More, larger parts in flight improve throughput up to your bandwidth and memory limits.

Q: Can a browser do a multipart upload?

Yes. Your backend generates presigned URLs for each UploadPart (and for create/complete), and the browser uploads parts directly. This combines multipart with presigning for large client-side uploads.

Q: Does multipart work for downloads too?

Yes. boto3's download_file fetches ranges in parallel using TransferConfig, and in SDK v3 you request byte ranges via GetObjectCommand's Range parameter to parallelize a large download.

Q: How do I show upload progress?

Pass a Callback to boto3's upload_file, or listen to the httpUploadProgress event on the v3 Upload object. Both report bytes transferred as parts complete.

Q: upload_file vs upload_fileobj in boto3?

upload_file takes a filesystem path; upload_fileobj takes a readable file-like object, which is what you want for in-memory buffers or streamed data. Both use the same multipart machinery.

A single PutObject maxes out at 5 GB and sends the whole body in one stream. For a large file that is both a hard limit and a reliability problem - one dropped connection restarts the entire transfer.

Multipart upload fixes this. It splits an object into parts, uploads them in parallel with independent retries, and assembles them server-side into one object. You rarely orchestrate that by hand: the boto3 transfer manager and the SDK v3 Upload helper do it for you from a single call. This page covers both the managed helpers and the low-level API underneath them.

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
s3 = boto3.client("s3")
# upload_file automatically uses multipart above the default 8 MB threshold.
s3.upload_file("big-video.mp4", "my-bucket", "videos/big-video.mp4")

// --- TypeScript (AWS SDK v3) ---
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";
 
const s3 = new S3Client({});
const upload = new Upload({
  client: s3,
  params: { Bucket: "my-bucket", Key: "videos/big-video.mp4", Body: createReadStream("big-video.mp4") },
});
await upload.done();

When to reach for this:

Uploading files larger than a few hundred megabytes, and mandatory above 5 GB.
Moving data over unreliable networks where a single-stream retry is expensive.
Streaming a body of unknown length (a pipe or generator) that has no Content-Length.
Wanting parallel throughput without writing part-management code.

Working Example

Upload a large file with tuned multipart settings and a progress callback, then confirm the result.

# --- Python (boto3) ---
import boto3
from boto3.s3.transfer import TransferConfig
 
s3 = boto3.client("s3")
 
# Tune the managed transfer: 16 MB parts, go multipart above 16 MB, 8 parallel parts.
config = TransferConfig(
    multipart_threshold=16 * 1024 * 1024,
    multipart_chunksize=16 * 1024 * 1024,
    max_concurrency=8,
    use_threads=True,
)
 
seen = {"bytes": 0}
def progress(n):
    seen["bytes"] += n
    print(f"uploaded {seen['bytes']} bytes")
 
s3.upload_file(
    "big-video.mp4", "my-bucket", "videos/big-video.mp4",
    Config=config, Callback=progress,
)
 
head = s3.head_object(Bucket="my-bucket", Key="videos/big-video.mp4")
print("done:", head["ContentLength"], "bytes")

// --- TypeScript (AWS SDK v3) ---
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";
 
const s3 = new S3Client({});
 
const upload = new Upload({
  client: s3,
  params: { Bucket: "my-bucket", Key: "videos/big-video.mp4", Body: createReadStream("big-video.mp4") },
  partSize: 16 * 1024 * 1024,   // 16 MB parts
  queueSize: 8,                 // 8 parallel parts
});
 
upload.on("httpUploadProgress", (p) => console.log(`uploaded ${p.loaded} bytes`));
await upload.done();
 
const head = await s3.send(new HeadObjectCommand({ Bucket: "my-bucket", Key: "videos/big-video.mp4" }));
console.log("done:", head.ContentLength, "bytes");

What this demonstrates:

TransferConfig (boto3) and the Upload options (v3) control part size, concurrency, and the multipart threshold.
Both helpers report progress via a callback or event as parts complete.
The helper creates, uploads, and completes the multipart upload for you - no manual part IDs.
A HeadObject afterward confirms the assembled object exists with the expected size.

Deep Dive

How It Works

Multipart has three phases: CreateMultipartUpload returns an UploadId, UploadPart sends each chunk and returns an ETag, and CompleteMultipartUpload stitches the parts (by number and ETag) into the final object.
Parts must be at least 5 MB each, except the last, and there is a maximum of 10,000 parts per object - which caps a single object at 5 TB.
The managed helpers pick a part size, upload parts in parallel up to the concurrency limit, retry failed parts individually, and call complete for you.
boto3 uses OS threads (use_threads=True) for parallelism; SDK v3 pipelines part uploads with a bounded queueSize.

The Low-Level API (Pause / Resume / Custom Orchestration)

Drop to the raw calls when you need to persist an UploadId across processes, resume after a crash, or upload parts from multiple machines.

# --- Python (boto3) ---
import boto3
 
s3 = boto3.client("s3")
bucket, key = "my-bucket", "videos/big-video.mp4"
 
create = s3.create_multipart_upload(Bucket=bucket, Key=key)
upload_id = create["UploadId"]
 
parts = []
with open("big-video.mp4", "rb") as f:
    part_number = 1
    while chunk := f.read(16 * 1024 * 1024):   # 16 MB parts
        r = s3.upload_part(Bucket=bucket, Key=key, PartNumber=part_number,
                           UploadId=upload_id, Body=chunk)
        parts.append({"PartNumber": part_number, "ETag": r["ETag"]})
        part_number += 1
 
s3.complete_multipart_upload(
    Bucket=bucket, Key=key, UploadId=upload_id,
    MultipartUpload={"Parts": parts},
)

// --- TypeScript (AWS SDK v3) ---
import {
  S3Client, CreateMultipartUploadCommand, UploadPartCommand, CompleteMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { readFileSync } from "node:fs";
 
const s3 = new S3Client({});
const Bucket = "my-bucket", Key = "videos/big-video.mp4";
 
const { UploadId } = await s3.send(new CreateMultipartUploadCommand({ Bucket, Key }));
 
const data = readFileSync("big-video.mp4");
const partSize = 16 * 1024 * 1024;   // 16 MB parts
const parts = [];
for (let i = 0, n = 1; i < data.length; i += partSize, n++) {
  const r = await s3.send(new UploadPartCommand({
    Bucket, Key, UploadId, PartNumber: n, Body: data.subarray(i, i + partSize),
  }));
  parts.push({ PartNumber: n, ETag: r.ETag });
}
 
await s3.send(new CompleteMultipartUploadCommand({
  Bucket, Key, UploadId, MultipartUpload: { Parts: parts },
}));

Persist the UploadId and completed part list, and you can resume a stalled upload by uploading only the remaining part numbers before calling complete.

Downloads

The same managers handle large downloads. boto3's download_file (also TransferConfig-driven) fetches ranges in parallel, and in SDK v3 you request byte ranges with GetObjectCommand's Range parameter or stream the body directly.

Gotchas

Orphaned parts keep billing - a failed or abandoned multipart upload leaves parts in storage that you pay for but cannot see with ListObjects. Fix: call AbortMultipartUpload on failure, and add a lifecycle rule with AbortIncompleteMultipartUpload to sweep them.
Part size below 5 MB - any part except the last must be at least 5 MB, or CompleteMultipartUpload fails. Fix: keep multipart_chunksize/partSize at 5 MB or higher.
Too many parts - very small parts on a huge file can exceed the 10,000-part limit. Fix: scale part size with file size so the count stays under 10,000.
Losing an ETag - complete requires each part's number and returned ETag, in order. Fix: collect {PartNumber, ETag} as you go and sort by part number before completing.
Assuming upload_file streams from memory - boto3's upload_file takes a path; upload_fileobj takes a file-like object. Fix: use upload_fileobj for in-memory or piped data.
No Content-Length on a stream - single PutObject needs a known length; a stream of unknown size does not have one. Fix: use the multipart helper, which does not require the total length up front.

Alternatives

Alternative	Use When	Don't Use When
Managed transfer helper (this page)	Large files, parallel throughput, minimal code	Tiny objects where a single PutObject is simpler
Single `PutObject`	Objects under a few hundred MB with known length	Files over 5 GB, or unreliable networks
Low-level multipart API	Pause/resume, distributed part uploads	Ordinary uploads the helper already handles
Presigned multipart URLs	A browser must upload a large file directly	Your server already has the bytes
S3 Transfer Acceleration	Uploading across long distances to a bucket	Same-region transfers where it adds cost, not speed

FAQs

When does multipart kick in automatically?

With the managed helpers, above a threshold - 8 MB by default in boto3's TransferConfig, and configurable in SDK v3's Upload via partSize. Below the threshold the helper does a single PutObject.

What is the maximum object size?

5 TB, assembled from up to 10,000 parts. A single non-multipart PutObject is capped at 5 GB, which is the practical trigger to switch to the transfer helper.

Do I ever need the low-level create/upload/complete calls?

Only for advanced control: resuming an interrupted upload across processes, uploading parts from multiple machines, or custom orchestration. For normal large uploads the managed helper is enough.

How do I tune throughput?

Raise concurrency (max_concurrency in boto3, queueSize in v3) and part size for big files on fast links. More, larger parts in flight improve throughput up to your bandwidth and memory limits.

Why am I being billed for uploads I never finished?

Incomplete multipart uploads leave parts stored but hidden from ListObjects. Abort them on failure and add a lifecycle rule with AbortIncompleteMultipartUpload to clean up automatically.

What is the minimum part size?

5 MB for every part except the last. Smaller non-final parts cause CompleteMultipartUpload to fail. Keep your chunk size at or above 5 MB.

Can a browser do a multipart upload?

Yes. Your backend generates presigned URLs for each UploadPart (and for create/complete), and the browser uploads parts directly. This combines multipart with presigning for large client-side uploads.

Does multipart work for downloads too?

Yes. boto3's download_file fetches ranges in parallel using TransferConfig, and in SDK v3 you request byte ranges via GetObjectCommand's Range parameter to parallelize a large download.

How do I show upload progress?

Pass a Callback to boto3's upload_file, or listen to the httpUploadProgress event on the v3 Upload object. Both report bytes transferred as parts complete.

upload_file vs upload_fileobj in boto3?

upload_file takes a filesystem path; upload_fileobj takes a readable file-like object, which is what you want for in-memory buffers or streamed data. Both use the same multipart machinery.

S3 Advanced Basics - the bulk list and delete operations around uploads.
Presigned URLs for Uploads & Downloads - let clients upload large files directly.
Lifecycle Rules & Storage Class Transitions - auto-abort incomplete multipart uploads.
Amazon S3 via SDK: Buckets & Objects - the single-object put/get baseline.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Recipe

Quick-reference recipe card - copy-paste ready.

# --- Python (boto3) ---
import boto3
 
s3 = boto3.client("s3")
# upload_file automatically uses multipart above the default 8 MB threshold.
s3.upload_file("big-video.mp4", "my-bucket", "videos/big-video.mp4")

// --- TypeScript (AWS SDK v3) ---
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";
 
const s3 = new S3Client({});
const upload = new Upload({
  client: s3,
  params: { Bucket: "my-bucket", Key: "videos/big-video.mp4", Body: createReadStream("big-video.mp4") },
});
await upload.done();

When to reach for this:

Uploading files larger than a few hundred megabytes, and mandatory above 5 GB.
Moving data over unreliable networks where a single-stream retry is expensive.
Streaming a body of unknown length (a pipe or generator) that has no Content-Length.
Wanting parallel throughput without writing part-management code.

Working Example

Upload a large file with tuned multipart settings and a progress callback, then confirm the result.

# --- Python (boto3) ---
import boto3
from boto3.s3.transfer import TransferConfig
 
s3 = boto3.client("s3")
 
# Tune the managed transfer: 16 MB parts, go multipart above 16 MB, 8 parallel parts.
config = TransferConfig(
    multipart_threshold=16 * 1024 * 1024,
    multipart_chunksize=16 * 1024 * 1024,
    max_concurrency=8,
    use_threads=True,
)
 
seen = {"bytes": 0}
def progress(n):
    seen["bytes"] += n
    print(f"uploaded {seen['bytes']} bytes")
 
s3.upload_file(
    "big-video.mp4", "my-bucket", "videos/big-video.mp4",
    Config=config, Callback=progress,
)
 
head = s3.head_object(Bucket="my-bucket", Key="videos/big-video.mp4")
print("done:", head["ContentLength"], "bytes")

// --- TypeScript (AWS SDK v3) ---
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { createReadStream } from "node:fs";
 
const s3 = new S3Client({});
 
const upload = new Upload({
  client: s3,
  params: { Bucket: "my-bucket", Key: "videos/big-video.mp4", Body: createReadStream("big-video.mp4") },
  partSize: 16 * 1024 * 1024,   // 16 MB parts
  queueSize: 8,                 // 8 parallel parts
});
 
upload.on("httpUploadProgress", (p) => console.log(`uploaded ${p.loaded} bytes`));
await upload.done();
 
const head = await s3.send(new HeadObjectCommand({ Bucket: "my-bucket", Key: "videos/big-video.mp4" }));
console.log("done:", head.ContentLength, "bytes");

What this demonstrates:

TransferConfig (boto3) and the Upload options (v3) control part size, concurrency, and the multipart threshold.
Both helpers report progress via a callback or event as parts complete.
The helper creates, uploads, and completes the multipart upload for you - no manual part IDs.
A HeadObject afterward confirms the assembled object exists with the expected size.

Deep Dive

How It Works

Multipart has three phases: CreateMultipartUpload returns an UploadId, UploadPart sends each chunk and returns an ETag, and CompleteMultipartUpload stitches the parts (by number and ETag) into the final object.
Parts must be at least 5 MB each, except the last, and there is a maximum of 10,000 parts per object - which caps a single object at 5 TB.
The managed helpers pick a part size, upload parts in parallel up to the concurrency limit, retry failed parts individually, and call complete for you.
boto3 uses OS threads (use_threads=True) for parallelism; SDK v3 pipelines part uploads with a bounded queueSize.

The Low-Level API (Pause / Resume / Custom Orchestration)

Drop to the raw calls when you need to persist an UploadId across processes, resume after a crash, or upload parts from multiple machines.

# --- Python (boto3) ---
import boto3
 
s3 = boto3.client("s3")
bucket, key = "my-bucket", "videos/big-video.mp4"
 
create = s3.create_multipart_upload(Bucket=bucket, Key=key)
upload_id = create["UploadId"]
 
parts = []
with open("big-video.mp4", "rb") as f:
    part_number = 1
    while chunk := f.read(16 * 1024 * 1024):   # 16 MB parts
        r = s3.upload_part(Bucket=bucket, Key=key, PartNumber=part_number,
                           UploadId=upload_id, Body=chunk)
        parts.append({"PartNumber": part_number, "ETag": r["ETag"]})
        part_number += 1
 
s3.complete_multipart_upload(
    Bucket=bucket, Key=key, UploadId=upload_id,
    MultipartUpload={"Parts": parts},
)

// --- TypeScript (AWS SDK v3) ---
import {
  S3Client, CreateMultipartUploadCommand, UploadPartCommand, CompleteMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { readFileSync } from "node:fs";
 
const s3 = new S3Client({});
const Bucket = "my-bucket", Key = "videos/big-video.mp4";
 
const { UploadId } = await s3.send(new CreateMultipartUploadCommand({ Bucket, Key }));
 
const data = readFileSync("big-video.mp4");
const partSize = 16 * 1024 * 1024;   // 16 MB parts
const parts = [];
for (let i = 0, n = 1; i < data.length; i += partSize, n++) {
  const r = await s3.send(new UploadPartCommand({
    Bucket, Key, UploadId, PartNumber: n, Body: data.subarray(i, i + partSize),
  }));
  parts.push({ PartNumber: n, ETag: r.ETag });
}
 
await s3.send(new CompleteMultipartUploadCommand({
  Bucket, Key, UploadId, MultipartUpload: { Parts: parts },
}));

Persist the UploadId and completed part list, and you can resume a stalled upload by uploading only the remaining part numbers before calling complete.

Downloads

Gotchas

Orphaned parts keep billing - a failed or abandoned multipart upload leaves parts in storage that you pay for but cannot see with ListObjects. Fix: call AbortMultipartUpload on failure, and add a lifecycle rule with AbortIncompleteMultipartUpload to sweep them.
Part size below 5 MB - any part except the last must be at least 5 MB, or CompleteMultipartUpload fails. Fix: keep multipart_chunksize/partSize at 5 MB or higher.
Too many parts - very small parts on a huge file can exceed the 10,000-part limit. Fix: scale part size with file size so the count stays under 10,000.
Losing an ETag - complete requires each part's number and returned ETag, in order. Fix: collect {PartNumber, ETag} as you go and sort by part number before completing.
Assuming upload_file streams from memory - boto3's upload_file takes a path; upload_fileobj takes a file-like object. Fix: use upload_fileobj for in-memory or piped data.
No Content-Length on a stream - single PutObject needs a known length; a stream of unknown size does not have one. Fix: use the multipart helper, which does not require the total length up front.

Alternatives

Alternative	Use When	Don't Use When
Managed transfer helper (this page)	Large files, parallel throughput, minimal code	Tiny objects where a single PutObject is simpler
Single `PutObject`	Objects under a few hundred MB with known length	Files over 5 GB, or unreliable networks
Low-level multipart API	Pause/resume, distributed part uploads	Ordinary uploads the helper already handles
Presigned multipart URLs	A browser must upload a large file directly	Your server already has the bytes
S3 Transfer Acceleration	Uploading across long distances to a bucket	Same-region transfers where it adds cost, not speed

FAQs

When does multipart kick in automatically?

What is the maximum object size?

5 TB, assembled from up to 10,000 parts. A single non-multipart PutObject is capped at 5 GB, which is the practical trigger to switch to the transfer helper.

Do I ever need the low-level create/upload/complete calls?

Only for advanced control: resuming an interrupted upload across processes, uploading parts from multiple machines, or custom orchestration. For normal large uploads the managed helper is enough.

How do I tune throughput?

Raise concurrency (max_concurrency in boto3, queueSize in v3) and part size for big files on fast links. More, larger parts in flight improve throughput up to your bandwidth and memory limits.

Why am I being billed for uploads I never finished?

Incomplete multipart uploads leave parts stored but hidden from ListObjects. Abort them on failure and add a lifecycle rule with AbortIncompleteMultipartUpload to clean up automatically.

What is the minimum part size?

5 MB for every part except the last. Smaller non-final parts cause CompleteMultipartUpload to fail. Keep your chunk size at or above 5 MB.

Can a browser do a multipart upload?

Does multipart work for downloads too?

Yes. boto3's download_file fetches ranges in parallel using TransferConfig, and in SDK v3 you request byte ranges via GetObjectCommand's Range parameter to parallelize a large download.

How do I show upload progress?

Pass a Callback to boto3's upload_file, or listen to the httpUploadProgress event on the v3 Upload object. Both report bytes transferred as parts complete.

upload_file vs upload_fileobj in boto3?

upload_file takes a filesystem path; upload_fileobj takes a readable file-like object, which is what you want for in-memory buffers or streamed data. Both use the same multipart machinery.

S3 Advanced Basics - the bulk list and delete operations around uploads.
Presigned URLs for Uploads & Downloads - let clients upload large files directly.
Lifecycle Rules & Storage Class Transitions - auto-abort incomplete multipart uploads.
Amazon S3 via SDK: Buckets & Objects - the single-object put/get baseline.

Stack versions: This page was written for boto3 1.43.x (Python 3.10+) and the AWS SDK for JavaScript v3 (Node.js 18+).

Multipart Upload & the Transfer Manager

Buscar en la documentación

Multipart Upload & the Transfer Manager