Design file storage and sync like Google Drive
Drive-style services store files and keep every device in sync. We design chunked storage, a metadata service and conflict handling.
Goal: users upload files, access them from any device, share them and have edits synced automatically.
Requirements
- Upload, download, edit and delete files; large files too.
- Sync across devices, including offline edits.
- Sharing with permissions; version history.
- Durable: files must never be lost.
Estimates
If 50 million users store 10 GB each, that is 500 PB. Storage and bandwidth are the cost drivers, so we want to avoid uploading unchanged data.
Split data and metadata
- File content in object storage, as chunks.
- Metadata (names, folders, owners, permissions, versions, chunk lists) in a database behind a metadata service.
Chunking
Break each file into chunks (for example 4 MB), each identified by a hash of its content.
- Resumable uploads: a failed upload resumes at the missing chunk.
- Delta sync: when a file changes, upload only the changed chunks, not the whole file.
- Deduplication: if a chunk hash already exists, don't store it again, which saves a lot of space across versions and users.
A file version is then just an ordered list of chunk hashes in the metadata.
Upload flow
- Client computes chunk hashes and asks the metadata service which are missing.
- It uploads missing chunks directly to object storage (presigned URLs).
- It commits the new file version in metadata (an atomic update of the chunk list).
- Other devices are notified.
Sync
Each device keeps a local database of file states and a cursor (last seen change). It subscribes to changes (long polling or WebSocket), fetches the list of changes after its cursor and downloads the needed chunks. A local watcher detects file edits and triggers uploads.
Conflicts
Two devices edit the same file offline. A common safe approach: keep both, saving one as "file (conflicted copy)", and let the user merge. Real-time collaborative documents use a different technique (operational transforms or CRDTs).
Sharing and security
Access control lists per file or folder, shareable links with expiry, encryption in transit and at rest, and virus scanning of uploads. Check permissions on every request, never trust the file ID alone.
Other concerns
Versioning and trash (recover deleted files for N days), quotas, and tiering older data to cheaper storage.
Takeaway: store chunks by hash in object storage, keep a transactional metadata service, upload only what changed, and sync with a change cursor.
System Design series, Part 3 (case studies): 10 of 16. Previous: Design search autocomplete (typeahead).
