Case study

Design Dropbox File Sync

Sync files and folders across devices with block-level deduplication and offline conflict handling.

syncchunkingconflict resolution

Requirements

Functional: upload/download files; folder hierarchy; sync across N devices; share links
Block-level dedup: identical chunks stored once
Offline edits merge on reconnect; conflict copies for same-file concurrent edits
Non-functional: sync latency < 30 sec for 10 MB file; 99.9% durability
Scale: 500M users; avg 1 GB stored; 10M concurrent sync clients

Back-of-envelope Estimation

Metric	Calculation	Result
Total storage	500M × 1 GB	500 EB order — dedup reduces ~3× → ~170 EB logical
Block size	4 MB chunks	Industry standard (Dropbox uses 4 MB)
Upload QPS	10M active × 2 files/day / 86,400	~230/sec avg
Metadata ops	10× upload rate	~2.3K/sec (list dir, sync cursor)
Block dedup savings	30% identical blocks	Massive storage reduction

API Design

Endpoint	Description
POST /v1/blocks/{hash}	Upload block if not exists (content-addressed)
GET /v1/blocks/{hash}	Download block
POST /v1/files/commit	{ path, blocks[], rev, clientRev } — atomic metadata commit
GET /v1/sync/delta?cursor=	Long-poll or cursor-based changes since last sync
GET /v1/files/{path}	Metadata + block list for file

Data Model

Entity	Key	Fields
blocks	content_hash (SHA-256)	size, s3_key, ref_count
file_metadata	namespace_id + path	block_hashes[], rev, modified_at, is_dir
sync_cursors	device_id	cursor_token, last_sync_at
namespaces	user_id or team_id	quota, root_rev

High-level Design

Sync architecture

Client ──► Sync API ──► Metadata DB (PostgreSQL)
              │              └──► Block Store (S3, keyed by hash)
              │
              └──► Local: chunk file ──► hash blocks ──► upload missing blocks only
              
Notification: change log ──► Kafka ──► push to connected devices (WebSocket/long-poll)

Deep Dive: Block Upload & Dedup

Client splits file into 4 MB blocks; computes SHA-256 per block
Batch check which hashes exist (POST /blocks/check)
Upload only missing blocks (content-addressed — idempotent)
Commit file metadata with optimistic locking on rev; 409 → conflict handler

Rolling hash optimization

rsync-style rolling hash finds moved blocks within large files without re-uploading unchanged sections.

Deep Dive: Conflict Resolution

Each file has monotonic server rev. Client sends base_rev on commit. If base_rev ≠ current rev, conflict: auto-merge for text (operational transform optional); otherwise create 'file (conflicted copy).ext' and notify user.

Delete tombstones propagate via delta feed
Folder moves are metadata-only operations (cheap)
ref_count on blocks enables GC when file deleted

Failure Modes & Monitoring

SLO	Target
Block durability	11 nines (S3)
Delta sync p99	< 5 sec
Conflict rate	Track; alert if > 1% commits

Monitor: block upload failures, sync cursor lag, orphaned blocks, ref_count leaks
GC job reclaims blocks with ref_count=0 after 30-day grace

System Design