Elasticsearch Search in Practice: When and How to Use It

1. The Role of ES: Solving the "Search No Longer Scales" Problem

First, let us clarify what Elasticsearch, referred to below as ES, actually is. It is a distributed full-text search and analytics engine built on Lucene. The two keywords are distributed and full-text search + analytics.

It is not a relational database, a KV store, or a replacement for your primary database. Its core use case is: when your data has reached the point where SQL LIKE and simpler solutions can no longer cope, ES is a heavyweight tool for fast search, complex filtering, and statistical aggregation across vast amounts of text.

Typical real-world pain points include:

• Product search: Millions of SKUs that must be searched by title, brand, category, attributes, and price range, with typo tolerance, highlighting, and relevance ranking.
• Log search: Hundreds of millions of log entries per day that must be located quickly by keyword, time range, and service name, then visualized in statistical charts.
• Content search: Millions of articles, posts, and comments requiring full-text search + tag filtering + sorting by likes or time.

In these scenarios, MySQL / PostgreSQL queries such as LIKE '%关键词%' will make your life miserable: they are slow, cannot use normal indexes, do not support Chinese tokenization, provide no relevance scoring, and make aggregation even more painful. ES was built for exactly these workloads.

2. Core Concepts: Mapping the Terminology to SQL

To understand ES, first align its terminology with SQL. It immediately becomes more familiar:

Sharding is the heart of ES's distributed architecture. The data in an index is divided into N primary shards distributed across multiple nodes, and each primary shard can have several replica shards. This lets the cluster scale horizontally to store more data, execute queries in parallel for better performance, and avoid data loss when a node fails. Note that the number of primary shards is fixed when the index is created. Changing it later requires reindexing, so leave enough headroom during planning.

The inverted index is the fundamental reason full-text search is fast. Traditional databases use a "forward index," finding content by document ID. An inverted index reverses that relationship by maintaining a list of every document in which each term occurs. When searching for "phone," ES does not scan every document. It looks up the list for "phone" and immediately obtains every relevant document ID. It then uses the BM25 algorithm to score and rank results by relevance.

3. Why ES Is Stronger Than SQL Full-Text Search

PostgreSQL has tsvector + GIN indexes, and MySQL has FULLTEXT indexes, so why is ES still more powerful? There are six dimensions:

1. Distributed architecture: Data is sharded by design. Add machines to handle more data and higher QPS. Horizontal scaling with SQL is painfully difficult.
2. Inverted indexes: Built specifically for search, with efficient matching plus phrase, prefix, fuzzy, and synonym searches.
3. Relevance scoring: The BM25 algorithm identifies which result is more relevant. SQL LIKE can only say "match / no match."
4. Aggregation capabilities: Bucket by fields, calculate metrics, and nest aggregations, producing statistics from hundreds of millions of records in seconds. This is ES's second major advantage.
5. Near real time: Data is searchable one second after being written by default, which works well for logging and monitoring.
6. Mature ecosystem: Kibana provides visualization, Logstash/Beats handles collection, and ELK offers an integrated toolchain.

In short, SQL means "store the data first and search it as a secondary capability," while ES is "built for search and analytics."

4. Query DSL Essentials: Do Not Start with match_all

ES describes queries in JSON through its Query DSL. The easiest mistake for beginners is failing to understand the difference between query and filter.

First, remember one hard rule:

• query context: Calculates relevance scores (_score) and answers "which result is most relevant?" A typical example is match.
• filter context: Does not calculate scores and only answers "match / no match." Its results are cached. Typical examples are term and range. Use a filter whenever possible because it is faster and consumes less CPU.

A practical DSL example for product search: match "wireless mouse" in the title, limit the price to 50–200, include the Logitech or Xiaomi brands, sort by relevance, and aggregate statistics by brand:

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "无线鼠标" } }
      ],
      "filter": [
        { "range": { "price": { "gte": 50, "lte": 200 } } },
        { "terms": { "brand": ["logitech", "xiaomi"] } }
      ]
    }
  },
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand", "size": 10 }
    }
  },
  "highlight": {
    "fields": { "title": {} }
  }
}

Several frequently used principles:

• Use match for full-text search because it analyzes text. Use term for exact matching because it does not analyze text and is commonly used with keywords, numbers, and booleans.
• Use bool to combine conditions through must (AND + scoring), should (OR), must_not (NOT), and filter (AND + no scoring + caching).
• Do not use term directly on a text field. The analyzer splits the field, so the query will probably return nothing. Use the .keyword subfield for exact matching.
• Aggregations (aggs) are returned independently of the query, allowing search and statistical reporting in the same request. This is cumbersome to implement in SQL.
• Do not overuse prefix wildcards such as *关键词 with wildcard. They are performance killers and should be avoided in production whenever possible.

5. Chinese Tokenization: Without IK, You Are Flying Blind

ES's default standard analyzer handles Chinese poorly. It splits the Chinese phrase "人工智能" ("artificial intelligence") into the individual characters "人," "工," "智," and "能," producing terrible search quality. For Chinese search, the IK analyzer is standard equipment.

IK provides two modes:

• ik_smart: Performs minimal segmentation and is suitable for use at search time, providing precision while avoiding excessive recall.
• ik_max_word: Performs the most fine-grained segmentation and is suitable for use at index time, maximizing recall so that more terms remain searchable.

Configure the mapping like this when creating an index:

PUT /articles
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      },
      "content": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      }
    }
  }
}

This is the classic combination for Chinese search: fine-grained segmentation at index time and coarse-grained segmentation at search time. Add custom dictionaries for trending words, new terms, and industry terminology to improve recall further. New terms can be added through IK's hot-update API without restarting the cluster. Starting a Chinese ES project without IK is effectively paying for an avoidable lesson.

6. When to Use ES

Do not adopt ES merely to show off. It has clearly defined, high-value use cases:

1. Product / content search: Large datasets requiring full-text search + multidimensional filtering + relevance ranking, as in e-commerce and content communities.
2. Logs and observability: ELK is the de facto standard. Fast search and aggregate analysis across massive log volumes are indispensable for troubleshooting.
3. When PostgreSQL / MySQL full-text search can no longer cope: The dataset exceeds ten million records, queries are complex, aggregations are frequent, and SQL is already overwhelming the primary database.
4. Vector search + hybrid search: Newer ES versions support dense_vector. For AI semantic search, they can combine "keyword + vector" search in one system, avoiding a separate vector database.

The decision rule is simple: if any one of "slow search, inaccurate search, or stalled analytics" applies, it is time to evaluate ES seriously.

7. When Not to Use ES

Using a heavyweight tool for a small problem costs more than money; it also creates operational and cognitive overhead. Avoid ES in these situations:

• Small datasets, within a few hundred thousand records: PostgreSQL full-text search + a GIN index is entirely sufficient, with lower latency and development cost.
• Strong transactions and strong consistency: ES writes are near real time, with a one-second refresh interval, and transactional support is absent. It is not a primary database. Do not store orders or accounts in it.
• No dedicated operations expertise on the team: ES cluster tuning, version upgrades, node scaling, and disk/memory planning all require experience. If nobody understands it, do not adopt it; troubleshooting failures can be extraordinarily difficult.
• Only simple CRUD + occasional LIKE queries: When their indexes are used correctly, PostgreSQL / MySQL can handle far more than you might expect.

Remember: the first principle of architecture selection is not "can it work?" but "is it worth it?"

8. The Cost of ES: Understand It Before Committing

ES is not a silver bullet, and its costs are substantial:

1. High resource consumption: It is a JVM application with a large appetite for memory. The recommended heap size is no more than half of physical memory, leaving the other half for Lucene's off-heap cache. Disk I/O is also significant, making SSDs the baseline.
2. Operational complexity: Cluster health, shard balancing, replica configuration, mapping changes that require reindexing when field types change, version upgrades, snapshot backups, and more all have their own pitfalls.
3. Eventually consistent, not a primary database: Data must be synchronized from PostgreSQL/MySQL through Canal, Logstash, Flink CDC, or application-level dual writes. Choose one. But once the synchronization pipeline fails, ES data diverges from the primary database, and troubleshooting is not easy.
4. Version fragmentation and major-version upgrades: ES 6 → 7 → 8 introduced multiple breaking changes, including removing types, enabling security by default, API changes, and the appearance of Serverless offerings. Cross-version upgrades are genuinely difficult, and many teams remain stuck on old versions.

In short: while using ES to solve a search problem, you also introduce a "heavyweight middleware system" that requires ongoing care and feeding.

Summary

• ES is a distributed full-text search + analytics engine whose primary domain is massive text search + complex filtering + aggregate statistics.
• Its three core components are inverted indexes for speed, shards / replicas for distribution, and relevance scoring for accuracy.
• Remember this rule for the Query DSL: query calculates scores; filter does not calculate scores and is cached. Use a filter whenever possible.
• Chinese search requires the IK analyzer. Using ik_max_word for indexing + ik_smart for searching is the standard combination.
• Use it for logs, product/content search, and workloads SQL can no longer handle. Do not use it for small datasets, strong transactions, or teams without operational expertise.

Key takeaway: ES is a heavyweight search tool, not a database. Use it as your primary database and the debt will eventually come due.