Design a web crawler

Goal: starting from seed URLs, download pages, extract links and follow them to collect content for a search index or analysis.

Requirements

• Scale to billions of pages; stay polite to websites.
• Avoid fetching the same page twice and avoid crawler traps.
• Respect robots.txt; handle failures and slow sites.
• Keep important pages fresh.

Main components

1. URL frontier: the queue of URLs waiting to be crawled.
2. Fetchers: workers that download pages.
3. DNS resolver (cached; DNS can be a bottleneck).
4. Parser: extracts text and links.
5. Deduplication: URL-seen check and content-seen check.
6. Storage: raw pages in object storage, metadata in a database.

The URL frontier

It is more than a queue. It must handle:

• Priority: important sites and frequently changing pages first.
• Politeness: one site must not receive hundreds of requests at once. Keep a separate queue per host, with a delay between requests to the same host, and obey Crawl-delay if given.

A common design maps each host to one worker (using hashing), so that worker controls the rate to that host.

Deduplication

• URL normalisation: lowercase the host, remove fragments, sort query parameters, resolve relative links.
• URL-seen set: a Bloom filter or large hash set, to skip already-known URLs.
• Content dedup: hash the page content (or use simhash for near-duplicates) to skip mirrors and copies.

Crawler traps

Infinite calendars, session IDs in URLs, and auto-generated link loops can trap a crawler. Defend with a maximum depth, a limit on URLs per domain, URL length limits and pattern detection.

robots.txt and ethics

Fetch and cache robots.txt for each host and obey it. Identify your crawler with a clear user-agent and contact URL. Avoid heavy load on small sites.

Distribution and failures

Split work across many machines by host hash. Retry failed fetches with backoff, set timeouts, and checkpoint frontier state so a crash does not lose progress.

Freshness

Re-crawl pages by how often they change: news homepages often, static pages rarely. Use HTTP ETag / If-Modified-Since to save bandwidth.

Content issues

Many pages need JavaScript rendering. A headless browser is slow and costly, so use it only where needed.

Takeaway: a per-host politeness-aware frontier, Bloom-filter URL dedup, content hashing, trap limits, and respect for robots.txt.

---

System Design series, Part 3 (case studies): 12 of 16. Previous: Design a ticket booking system without double booking.