Elasticsearch and full-text search: how the inverted index works
Searching text with LIKE is slow. Elasticsearch builds an inverted index for fast, ranked search. Learn how it works and where it fits.
A query like WHERE body LIKE '%payment%' scans every row and cannot rank results. Elasticsearch (built on Apache Lucene) is designed for full-text search.
The inverted index
Instead of "document, then words", it stores "word, then documents":
| Term | Documents |
|---|---|
| payment | 3, 8, 21 |
| refund | 8, 40 |
| delay | 3, 40 |
To find "payment refund", it looks up both terms and intersects the lists. This is fast even for millions of documents.
Analysis
Before indexing, text goes through an analyzer: split into words, lowercase, remove common words, and reduce words to a stem ("running" to "run"). The same steps are applied to the query, so "Refunds" matches "refund".
Relevance
Results are ranked by a score, commonly BM25, which considers how often a term appears in a document, how rare it is across all documents and the document length. You can boost fields, such as making a title match count more than a body match.
Distributed by design
An index is split into shards, spread across nodes, and each shard can have replicas. Searches run on all shards in parallel and the results are merged. Writes become searchable after a short refresh (about one second by default), so it is near real-time, not instant.
Common uses
Site search, log search (the ELK/Elastic stack), autocomplete, filtering and faceted navigation, and analytics on text.
What it is not
- It is not your primary database. Keep the source of truth in a relational or other store, and sync data to Elasticsearch.
- It has no multi-document transactions.
- Mapping mistakes and too many shards cause real pain, so plan them.
For a small site, the search built into PostgreSQL or SQLite can be enough before you add a separate cluster.
Takeaway: use an inverted-index engine for text search and ranking, keep your main database as the source of truth, and keep the index in sync.
System Design series, Part 2 (building blocks): topic 4 of 14. Previous: RabbitMQ explained: exchanges, queues and routing.
