Log Aggregation Architecture: ELK vs. Loki

At 2 a.m., the payment service starts throwing errors. You SSH into the first machine and run tail -f: nothing. The second machine: nothing. The third machine... After checking twenty machines, you discover that the critical log was on the seventeenth, which the autoscaler replaced ten seconds ago. Every operations engineer has lived through some version of this scenario: services run on different machines and in different containers, logs are scattered everywhere, and the evidence is impossible to find during an incident.

Log aggregation solves this problem by centralizing logs from every machine and container in one place where they can be searched, queried, and used for alerts. This article explains its four-layer architecture, the tradeoffs between the two mainstream options ELK and Loki, why structured logging is a prerequisite for everything else, and how to define a production-grade logging policy.

1. Four Architectural Layers: Collection → Transport → Storage → Query

Regardless of the chosen solution, log aggregation has the same four-layer foundation. Once you understand these layers, any logging system becomes easier to navigate.

┌─────────────┐    ┌──────────┐    ┌───────────────┐    ┌─────────────────────┐
│ Collection  │ →  │ Transport│ →  │ Storage/index │ →  │ Query/visualization │
│ filebeat    │    │ (Kafka)  │    │ Elasticsearch │    │ Kibana              │
│ promtail    │    │  buffer  │    │ / Loki        │    │ / Grafana           │
│ vector      │    │ optional │    │               │    │                     │
└─────────────┘    └──────────┘    └───────────────┘    └─────────────────────┘
 agent per host       absorb bursts      index/compress       investigate/alert

Collection layer: A lightweight agent runs on every machine or node, reading log files, tracking offsets, applying labels, and shipping records upstream. The ELK ecosystem uses Filebeat. Logstash is the veteran option, but it is too heavy and is now generally used only as a processing node. The Loki ecosystem uses Promtail. Another rising option is Vector, written in Rust with strong performance and unified configuration. A collector must provide two properties: no duplicates or gaps, by recording its read offset and resuming after a restart, and backpressure, so a blocked downstream system neither causes logs to be dropped nor exhausts the collector's resources.

Transport layer: Optional. For high log volumes or unstable downstream systems, Kafka can provide an intermediate buffer. Its benefit is absorbing bursts and smoothing throughput. If downstream storage fails for five minutes, logs accumulate in Kafka instead of being lost. Small deployments can skip this layer and push directly from collectors to storage. Many midsize teams are better off without Kafka because every additional component adds operational cost.

Storage and indexing layer: The core of the logging system and the point where ELK and Loki diverge. This layer determines how logs are stored, how fast queries run, and how much the system costs.

Query and visualization layer: The interface people ultimately use. Kibana pairs with ELK, while Grafana pairs with Loki. Both support writing queries, building dashboards, configuring alerts, and filtering by time and fields.

The next two sections examine how ELK and Loki differ at the storage and indexing layer, which is the decisive part of the architecture.

2. ELK: Heavy Artillery with Powerful Full-text Search

ELK = Elasticsearch + Logstash + Kibana. Filebeat has largely replaced Logstash, but the name remains. Its core is Elasticsearch, a Lucene-based search engine.

ES follows an all-field indexing philosophy. When each log arrives, ES builds an inverted index for every field and tokenizes the message body. Given {"msg":"order timeout","user_id":"8821"}, you can later query by user_id, search msg for "timeout," or combine them as user_id=8821 AND msg:"timeout", and each query returns quickly. Full-text search is ELK's killer feature: any word in a log message can be found.

The tradeoff is that it is heavy. Indexing every field consumes CPU, memory, and disk:

• Memory: ES keeps hot index data in memory through the JVM heap + OS page cache, often consuming tens of GB. An ES node is effectively unusable without 32GB of memory.
• Disk: The original logs plus their indexes commonly produce an expansion ratio between 1:1.5 and 1:3. Storing 1TB of logs can consume 2–3TB in practice.
• Operational complexity: An ES cluster requires management of shards, replicas, cluster health, and version upgrades, each with its own pitfalls. Too few shards constrain scaling; too many cause a metadata explosion.

ELK's advantages are its mature ecosystem and powerful features: aggregation and analytics, machine-learning anomaly detection, SIEM security analysis, APM, and more. The company behind ES has built this ecosystem for many years. If your primary requirement is complex full-text search and aggregation over log messages, ELK is the most reliable choice. Business-facing search for e-commerce products or document libraries is also ES's home turf; logging is simply another workload it handles well.

3. Loki: Lightweight and Label-only Indexing

Loki is a Grafana product with a philosophy exactly opposite to ELK: it does not index log messages, only labels.

As an analogy, ES "indexes every word in every log," while Loki "attaches a few labels such as service name, level, and Pod name, then stores the original message in compressed form." This closely resembles Prometheus for logs: Prometheus indexes only metric names + labels, not the values themselves. Loki applies the same idea to logs.

ELK:    [service=api][level=error] + full-text message index  ← heavy, but messages are searchable
Loki:   [service=api][level=error] + compressed message only ← light, queried by labels + time

The tradeoff is that searching log messages is impossible or difficult. A Loki query first uses labels and a time range to identify a group of log streams, then applies regular-expression filters to the results. When you search for the word error, Loki must retrieve all logs matching the labels before filtering them with a regular expression, which becomes slow at scale. Loki is therefore suitable for queries based on labels + time ranges, not for searching the entire corpus for a keyword.

The benefit is efficiency:

• Resource-efficient: Without all-field indexes, Loki uses far less CPU, memory, and disk than ES. At the same log volume, Loki's storage cost is often 1/5 to 1/10 of ELK's.
• Operationally simple: It can run as a single process. Production deployments should be distributed, but the conceptual burden remains low, without the complexity of ES shards and replicas.
• Integrated with Prometheus/Grafana: The label systems align naturally. In Grafana, you can drill directly from a metrics panel into logs. When p95 spikes, one click retrieves logs from that machine and time window. Reproducing this experience with ELK takes considerably more work.

Loki is suitable when most queries ask "what errors did this service report during this period?" or "what did this Pod log before restarting?" rather than "who mentioned timeout anywhere in the entire log corpus?" The latter is less common. Most investigations begin with a metric or alert, narrow the scope to a service and time range, and then inspect the logs.

4. ELK vs. Loki: How to Choose

Neither solution is universally better. The choice depends on your query patterns. Here is a practical comparison:

Practical recommendation: Most small and midsize teams should start with Loki. Around 90% of log queries filter by service + time + level. Loki handles these quickly and cheaply, with seamless Grafana integration. If a genuine need to search log messages later emerges, such as finding a keyword for a compliance audit or analyzing text in business logs, introduce ES for that subset. Do not deploy ELK from the outset for 10% of use cases while paying 100% of its cost.

Conversely, if search is the business itself, such as e-commerce, document libraries, or a log-analysis product, or logs require complex aggregation and analytics, ELK's full-text capabilities are irreplaceable. Use ELK directly instead of forcing regular expressions into Loki.

A common hybrid is to use Loki for high-volume operational logs with simple queries and ES for the smaller volume of business logs that require deeper analysis. Both systems coexist, each handling the workload it suits best.

5. Structured Logging (JSON): The Prerequisite for Everything

Whether you choose ELK or Loki, logs become useful only when they are structured as JSON. This must be implemented in the application before collection begins.

# Anti-pattern: pleasant for humans, difficult for machines to query
2026-07-18 10:23:11 ERROR User 8821 order failed: timeout calling payment

# Correct approach: JSON with indexable and filterable fields
{"ts":"2026-07-18T10:23:11.123Z","level":"error","msg":"order_failed",
 "service":"order-api","user_id":"8821","error":"timeout","downstream":"payment",
 "trace_id":"a3f9c2","duration_ms":8203}

Structured logging provides three benefits:

1. Machine-queryable: ELK and Loki can filter directly by user_id=8821 or downstream=payment, without using regular expressions to extract fields from a sentence. Unstructured logs either require Grok/LogQL regular-expression parsing, which is slow, brittle, and error-prone, or cannot be queried by field at all.
2. Trace integration: Store trace_id as a field, and logs can be linked to traces. You can jump from a trace to its logs or from a log to its trace, creating a complete investigation path.
3. Low collection cost: JSON can be serialized and transported directly, so the collector does not need to parse it. Promtail and Filebeat can both extract labels automatically from JSON fields.

Every service should emit logs as JSON. This is a team-level standard, not a personal preference. Use pino for Node, structlog or a JSON formatter for Python, and zap/slog for Go. Every language has mature structured-logging libraries, so there is no reason to keep writing sentences intended only for humans.

6. Logging Policy: Levels, Sampling, and Retention

Building a log aggregation system does not mean collecting every possible log. A production-grade logging policy must answer three questions: which levels to record, how much to sample, and how long to retain data.

Levels:

• debug: Used for development and troubleshooting; disabled by default in production or sampled at a very low rate.
• info: Important business events such as orders, payments, and logins, plus milestones along normal execution paths.
• warn: Abnormal but recoverable conditions such as successful retries, degradation, or triggered rate limits.
• error: A genuine failure requiring human intervention. Do not use error as a trash can. User validation failures and third-party 429 responses are not necessarily system errors. Logging them as errors buries genuine failures in noise.

Sampling: High-QPS paths cannot log every event. If a 100,000 QPS endpoint records an info log for every request, it generates 8.6 billion records per day, potentially costing more to store than the servers cost to run. A common policy is to sample 1% of healthy requests and retain 100% of abnormal requests. Loki can sample at the collection layer, while ES deployments generally control sampling in the application. Sampling is not indiscriminate data loss; it uses statistics to control cost. A 1% sample is enough to reconstruct the overall distribution, while no error should be missed.

Retention: Logs cannot be stored forever, so retention should be tiered by value:

• Hot data (7 days): Recent logs on SSD, optimized for fast queries and daily troubleshooting.
• Warm data (30 days): Moved to lower-cost disks or object storage for occasional historical investigations.
• Cold archive (90 days+): Compressed and stored in S3/OSS, rarely queried, and retained for compliance.

ES uses ILM (Index Lifecycle Management) for automatic tiering. Loki uses boltdb-shipper + an S3 backend and supports hot/cold tiering naturally. Retention periods must align with compliance requirements. Industries such as finance and healthcare impose minimum retention periods, so do not delete data early merely to reduce costs.

7. Logs, Metrics, and Traces: Complementary, Not Interchangeable

One final misconception needs clarification: does log aggregation eliminate the need for Prometheus? No. Logs, metrics, and traces are the three pillars of observability. Each serves a different purpose; they complement rather than replace one another.

• Logs (details): Discrete event records that explain "what happened in this specific instance." They are expensive and high-volume but suitable for deep investigation.
• Metrics (aggregated numbers): Time-series values that show "the overall trend." They are inexpensive and well suited to alerts and dashboards. Error rates, QPS, and p95 latency should come directly from Prometheus, not from counting logs.
• Traces (request paths): Tree-shaped records of one request across services that reveal "which hop was slow." They are useful for locating bottlenecks in distributed systems.

A typical workflow combines all three: a metric triggers an alert ("payment error rate increased") → a trace locates the problem ("the slow hop was the bank gateway call") → logs reveal the details ("that gateway request returned timeout"). Calculating metrics from logs is the most expensive approach. Scanning hundreds of millions of logs per second to calculate an error rate is far less efficient than having Prometheus store the time series directly. Logs are valuable for detail, not aggregation. Connect all three with trace_id to form a single network. That integration is "observability," not three isolated systems.

Summary

• Log aggregation provides centralized search: Services run on different machines, and hunting through logs over SSH belongs to a previous era. Collection → transport → storage → query is the common architecture underlying every logging system.
• ELK excels at full-text search but is expensive and heavy: ES indexes every field, making log messages searchable at the cost of high memory, disk, and operational overhead. It suits businesses built around search or scenarios requiring complex text analysis.
• Loki excels at efficiency but is weak at message search: It indexes only labels and integrates with Grafana, using a fraction of ELK's resources. It handles most operational log queries by service + time + level, so small and midsize teams should start with Loki.
• Structured logging (JSON) is the prerequisite for everything: It enables machine queries, trace integration, and low-overhead collection. Every service should emit JSON logs; this is a standard, not a preference.
• Policy determines cost: Do not use levels as trash cans, sample high-QPS paths, and retain data in hot/warm/cold tiers. Logging strategy affects cost more than tool selection does.
• Logs, metrics, and traces complement rather than replace one another: Logs provide details, metrics show trends, and traces identify request paths. Connect them with trace_id. Calculating metrics from logs is the most expensive anti-pattern.

In one sentence: The value of a log aggregation system does not depend on whether you chose ELK or Loki, but on whether you can retrieve the logs for "that machine, that time window, and that trace_id" within ten seconds of an incident. Tools are the means; queryability is the goal.