Logo

Chapter 2 Upload Path: Chunking, Concurrency & Resume (Key)

On the upload side, the goal is to get as close as possible to “saturating the bandwidth” under fluctuating networks, while limiting the cost of failures to the minimum (retransmit only failed chunks). Together, these two goals largely explain why uploads feel faster.

2.1 Chunking

  • The client MUST split the file into multiple chunks; a chunk is the minimum unit for upload/retry.
  • Chunks SHOULD use a fixed size (or a size-by-file policy) to simplify concurrency scheduling and state representation.
  • Immediate benefit: any failure affects only one chunk, not the entire file.

2.2 Parallel upload

The purpose of concurrency is not “more threads”, but to avoid three practical limitations of a single connection in real networks:

  • Slow start and window growth: a single connection ramps up slowly, especially with high cross-region RTT.
  • Link jitter: jitter on one connection can drop overall speed; concurrency smooths the effect.
  • Per-flow shaping: some networks throttle a single flow; multiple flows can get closer to the bandwidth ceiling (within compliance constraints).
  • The client SHOULD use limited concurrency (e.g., 4–12), and lower it on weaker devices to avoid memory spikes.
  • The client MAY adapt concurrency: increase when throughput rises; decrease when error rate rises.

2.3 Idempotency & retry

  • Repeated uploads of the same chunk MUST be idempotent: duplicate submissions must not corrupt an already-completed state.
  • The server SHOULD enforce uniqueness with (transferId, fileId, chunkIndex) or an equivalent constraint.
  • The client MUST implement exponential backoff retries; failures SHOULD retry only failed chunks.

2.4 Resumable upload

  • The server MUST be able to answer “which chunks are already uploaded”, so the client can skip completed ones.
  • The uploaded set SHOULD be represented as a bitmap / range-set to avoid state size growing linearly with chunk count.
  • Speed benefit: after network interruptions, browser crashes, or network switching, resumption does not restart from 0%.

2.5 “Share while uploading” (optional)

  • The system MAY generate a usable transfer identifier and share link while uploading is still in progress to accelerate delivery workflows.
  • If sharing is allowed before upload completion, it MUST be explicit what recipients see: incomplete content is not downloadable yet, or only progress is shown.