Design an online judge for coding contests

Goal: users submit code; the system runs it against hidden test cases and returns a verdict (Accepted, Wrong Answer, Time Limit, and so on) quickly.

Requirements

• Support many languages.
• Run untrusted code safely.
• Enforce time and memory limits.
• Handle contest spikes, when thousands submit at once.
• Show results and a ranked leaderboard.

Architecture

1. API server: receives the submission, stores it, returns an ID.
2. Queue: holds pending submissions.
3. Judge workers: take a submission, run it in a sandbox and record the verdict.
4. Database: problems, test cases, submissions, results.
5. Storage: test data in object storage, cached on workers.
6. Result delivery: the client polls, or gets results over WebSocket or SSE.

Why a queue

Judging takes seconds and varies in length. A queue smooths spikes, lets workers scale independently, and allows retries if a worker dies. Use separate queues or priorities, for example contest submissions before practice.

The sandbox (most important)

User code can try to read files, open network connections, fork-bomb or run forever. Isolate it:

• Containers or microVMs with a minimal filesystem.
• cgroups to limit CPU time, memory and process count.
• Seccomp / syscall filters to block dangerous system calls.
• No network access and a read-only filesystem except a small temporary folder.
• Run as an unprivileged user, and destroy the environment after each run.
• Wall-clock timeouts in addition to CPU limits.

Tools such as isolate or gVisor exist for this. Never run submitted code on the host directly.

Judging flow

1. Compile (if needed) with its own limits.
2. For each test case, feed input, capture output, enforce limits.
3. Compare to the expected output (exact match or a special checker for problems with many valid answers).
4. Stop at the first failure (or run all, depending on scoring) and return the verdict.

Accuracy and fairness

Time measurements vary by machine load, so run on dedicated, similar workers, and re-run borderline cases. Hide test data so users cannot hard-code answers.

Leaderboard

A Redis sorted set keeps ranks cheaply. Update it when a result is final.

Scaling

Add workers behind the queue. Pre-build language images, cache test data locally and keep a warm pool during contests.

Takeaway: queue submissions, run each in a locked-down sandbox with strict limits, compare with a checker, and scale workers independently.

---

System Design series, Part 3 (case studies): 15 of 16. Previous: Design a flash sale system that survives the rush.