Kafka and Stream Processing in Practice

First, a counterintuitive fact:

Many teams use Kafka as an "advanced version of RabbitMQ": send messages, receive messages, acknowledge them, and move on. Then one day, someone from the business asks, "Can we process user behavior in real time to build real-time profiles?" Everyone stares at each other, then starts setting up a separate Flink stack.

The root cause of this awkward situation is that Kafka has never been merely a message queue; it is a distributed event streaming platform. Using it only as an MQ is like buying an excavator to deliver takeout: it works, but wastes 90% of its capabilities while adding substantial operational overhead.

This article explains Kafka's underlying model, why its throughput is so high, how consumer groups work, how ordering and Exactly-Once guarantees are achieved, and when you genuinely need Kafka.

1. What Is Kafka? Not an MQ, but an Event Streaming Platform

To understand Kafka, start with the word "log."

Kafka's core data structure is not a queue, but an ordered log that supports append-only writes, replay, and offset-based reads. This design originated from LinkedIn's internal data infrastructure requirements: it needed to deliver activity data such as page views, monitoring metrics, and user behavior from producers to multiple consumers with high throughput, low latency, and replayability.

The difference between a queue and a log is fundamental:

Queue: consumption deletes the message; each message has only one lifetime
Log: consumption only advances the offset; messages remain until expiration and can be replayed multiple times

This single distinction determines all of Kafka's characteristics:

• Replayability: If a consumer fails, its logic changes, or a new downstream system is added, historical data can be consumed again. This is the foundation of stream processing.
• Independent subscribers: Different consumer groups maintain their own offsets without interfering with one another.
• High throughput: Sequential disk appends plus batched reads and writes treat disks as sequential devices.

Kafka solves three categories of problems:

1. Log aggregation: Collecting application logs, access logs, and monitoring metrics in one place
2. Event streams: A publish-subscribe bus for business events such as order placement, payment, and inventory changes
3. Event-driven systems / stream processing: Real-time ETL, real-time data warehouses, real-time risk control, and CDC synchronization

In one sentence: Kafka is a high-throughput, durable, replayable distributed log system built for event streams.

2. The Core Model: Topic / Partition / Offset / Consumer Group

Understand these four concepts together, and you understand more than half of Kafka.

Topic

A logical category of messages, such as order-events or user-clicks. For developers, it is simply the address that determines where to send data and where to read it from.

Partition: The Key to Parallelism

Each Topic is divided into multiple Partitions. Each Partition is an ordered, immutable, continuously appended log.

Topic: order-events  (3 partitions)

  P0: [m0][m1][m2][m3][m4] ...
  P1: [m0][m1][m2][m3] ...
  P2: [m0][m1][m2][m3][m4][m5] ...

A Partition is Kafka's unit of parallelism. Throughput scales horizontally by adding partitions: writes to one partition can only be handled by its leader on one Broker, but multiple partitions can be distributed across multiple Brokers to handle load in parallel.

Offset: The Consumption Position

Within each Partition, every message has a monotonically increasing integer offset. Consumers record progress by saying, "I have read through offset X."

Offsets are maintained by the consumers themselves and stored in Kafka's internal __consumer_offsets topic. This means consumers can move forward, move backward, or remain in place. Replaying history, skipping bad data, or rolling back to a point in time is simply a matter of changing an offset.

Consumer Group

This is what distinguishes Kafka from ordinary pub/sub systems. Within the same consumer group, a message is consumed by only one consumer, while different consumer groups each consume the complete stream independently.

A Topic has 3 partitions, and consumer group G1 has 3 consumers:
  P0 → C0
  P1 → C1
  P2 → C2

Add consumer C3 to G1? C3 remains idle because there are not enough partitions.
Add an independent consumer group G2? Consumers in G2 receive the complete message stream independently.

Kafka therefore supports two semantics at the same time:

• Queue semantics: Multiple consumers within one consumer group share the load to increase throughput
• Publish-subscribe semantics: Multiple consumer groups consume independently, providing broadcast behavior

3. Why Kafka Is So Fast: Sequential Writes, Zero-Copy, Batching, and Partition-Level Parallelism

A single Kafka machine can handle hundreds of thousands or even millions of TPS. This is not magic; it is the combined result of several sound design decisions.

1. Sequential Disk Writes

Random disk writes are as slow as they were in the 1990s, but sequential writes are extremely fast. In some scenarios, they are even faster than random memory access. Kafka only appends to logs and never modifies existing records, naturally matching the sequential characteristics of disks.

2. Batching and Compression

A Producer does not send every message immediately. It accumulates messages into batches before sending them. Batches can also use end-to-end compression with Snappy / LZ4 / Zstd, reducing both network and disk usage. For production throughput tuning, the key parameters are batch.size and linger.ms.

3. Zero-Copy (sendfile)

When consumers read messages, Kafka uses Linux's sendfile system call. Data travels directly from the page cache to the network interface through DMA, without entering user space at all.

Traditional: Disk → kernel page cache → user space → Socket buffer → NIC  (4 copies)
Zero-copy:   Page cache → NIC                                        (2 copies, performed by DMA)

When most consumer data is still in the page cache as hot data, Kafka is effectively forwarding data from memory to the network, delivering exceptional performance.

4. Partition-Level Parallelism

A single Partition has a throughput limit, primarily determined by sequential I/O capacity. Partitions, however, are horizontal shards, so three partitions can approximately triple throughput. Combined with replication, Kafka provides both high availability and scalability.

4. Consumer Groups and Partitions: Partitions Enable Parallelism, While Rebalancing Is a Double-Edged Sword

Remember this hard rule:

Within the same consumer group, a Partition can be consumed by only one consumer.

To increase consumption parallelism, you must add partitions; adding consumers alone is not enough. When the number of consumers exceeds the number of partitions, the additional consumers do nothing.

Partition count = 6 → at most 6 consumers in the group can work
Increase to 10 consumers? 4 will always remain idle

This is also why the partition count must be planned when creating a Topic. Adding partitions later disrupts key-routing consistency: the same key may be assigned to different partitions, breaking ordering.

Rebalance

Consumers within a consumer group change as instances scale out, scale in, fail, or experience network instability. Kafka must then reassign partitions among the surviving consumers. This process is called a rebalance.

A rebalance is a "stop the world" operation: while it is in progress, every consumer in the group pauses consumption. Frequent rebalances are one of the most common causes of Kafka consumer incidents:

Consumer processing slows down → heartbeat times out → consumer is removed → rebalance → backlog grows → processing becomes even slower → another timeout... an endless loop.

Countermeasures: Configure session.timeout.ms / heartbeat.interval.ms / max.poll.interval.ms appropriately. Increase max.poll.interval for long-running tasks, and do not let the processing time for a single batch exceed the heartbeat budget.

5. Ordering and Delivery Semantics: Ordering Within a Partition and Exactly-Once Guarantees

Ordering Guarantees

Kafka's ordering model can be summarized in one sentence: ordered within a partition, unordered across partitions.

If events for the same order must be processed sequentially, use orderId as the message key. The Producer hashes the key to a fixed partition, ensuring that identical keys enter the same partition and naturally preserve order.

Producer hashes the key → fixed Partition
  orderId=o1 → P0  (all events for o1 are arranged in write order)
  orderId=o2 → P1

Do not pursue global ordering. It requires using only one partition, which reduces parallelism to 1 and throughput along with it. Engineering systems almost never need global ordering; they need ordering at the business-entity level.

Delivery Semantics

Configuration determines three possible semantics:

At-Least-Once plus business-level idempotency is the most pragmatic engineering solution. This combination is enough for 99% of scenarios: duplicate messages are deduplicated using a unique business key, at far lower cost than introducing transactions.

Exactly-Once has been supported since Kafka 0.11 and requires:

1. An idempotent producer (enable.idempotence=true): The producer attaches a sequence number to each message, and the broker deduplicates messages, preventing duplicates caused by retries from a single producer
2. Transactions: The entire consume-process-produce chain is wrapped in a transaction so that it either succeeds completely or rolls back completely. This is primarily intended for Kafka Streams and scenarios that consume from Kafka and write back to Kafka

One key point deserves emphasis: Exactly-Once is a Kafka-internal semantic, not an end-to-end guarantee. If you consume from Kafka and write to MySQL, business-level idempotency is still required to guarantee exactly-once behavior. Do not be misled by marketing language.

6. Stream Processing: Kafka Streams and ksqlDB

Because Kafka is itself an event streaming platform, processing streams directly on top of it is a natural fit.

Kafka Streams

Kafka's built-in stream-processing library is a library, not an independent compute cluster. It runs inside your application process and provides a programming model for consuming from Kafka, transforming data, and writing it back to Kafka:

StreamsBuilder builder = new StreamsBuilder();
builder.stream("user-clicks")
       .filter((k, v) -> v.region().equals("cn"))
       .groupBy((k, v) -> v.userId())
       .count()
       .toStream()
       .to("user-click-counts");

Key characteristics: no independent cluster, native Exactly-Once support, RocksDB for state management, and changelog topics for fault tolerance. It is the first choice for lightweight stream processing and is far simpler than deploying Flink.

ksqlDB

If you want to write stream-processing logic in SQL, ksqlDB is the official answer:

CREATE STREAM clicks (userId VARCHAR, region VARCHAR)
  WITH (KAFKA_TOPIC='user-clicks', VALUE_FORMAT='JSON');

CREATE TABLE click_counts AS
  SELECT userId, COUNT(*) AS cnt
  FROM clicks WINDOW TUMBLING (SIZE 5 MINUTES)
  GROUP BY userId;

It is suitable for declarative scenarios such as real-time dashboards, simple ETL, monitoring, and alerting. Kafka Streams / Flink remains more flexible for complex windows and custom aggregations.

When should you choose Flink instead of Kafka Streams?

• You need complex window semantics such as session windows or late-data handling
• You need unified stream and batch processing, with the same code running in both modes
• You need to work across multiple data sources such as Kafka + databases + files
• You need unified scheduling for large-scale clusters

For simple real-time ETL that reads from Kafka and writes to Kafka, Kafka Streams offers better value.

7. When to Use Kafka and When Not To

This is the most important section. Many teams adopt Kafka because they have heard that large companies use it, not because they genuinely need it.

Use Kafka When

• Aggregating logs, metrics, and tracking events: Multiple data centers, many services, and terabytes of data are classic Kafka territory
• Building event-driven microservices: A business event bus with multiple downstream systems subscribing to the same event
• Building real-time data pipelines: CDC → Kafka → data warehouse / lake / Flink
• Processing streams: Real-time risk control, recommendations, and dashboards
• Replaying messages is required: Consumer logic changes or A/B recomputation is needed

Do Not Use Kafka When It Is the Wrong Tool

• Simple task queues: For sending emails or running scheduled tasks, RabbitMQ / Redis is enough
• Complex routing: Header-based routing, priority queues, and dead-letter recovery are where RabbitMQ excels
• Low-latency, tiny messages: Kafka is optimized for throughput, not latency. Using Kafka for messages only a few dozen bytes long is wasteful
• Small or medium workloads with limited operational capacity: Operating a Kafka cluster is not lightweight. Brokers, Zookeeper/KRaft, replicas, monitoring, and disk expansion all require work. A three-person team adopting Kafka will probably be overwhelmed by the operational burden

Quick Selection Guide

In one sentence: Do not use Kafka to solve RabbitMQ's problems, and do not expect RabbitMQ to sustain Kafka-level throughput.

Summary

Here are the key points:

1. Kafka is an event streaming platform, not merely a message queue. Its foundation is an append-only ordered log that supports replay and independent subscribers, which determines all its characteristics.
2. Partitions are the key to parallelism: Throughput scales by adding partitions, while consumption parallelism is limited by the partition count. Plan the partition count when creating the Topic.
3. Its speed is not magic, but the result of layered engineering: Sequential writes, batching, zero-copy, and partition-level parallelism together enable millions of TPS.
4. Use partition-level ordering for order guarantees, and At-Least-Once plus idempotency for delivery. Exactly-Once is a Kafka-internal semantic; end-to-end consistency still requires safeguards in business logic.
5. Kafka Streams suits lightweight ETL, while Flink suits complex computation. Do not reach for Flink immediately; get Kafka Streams running effectively first.

One final line to remember:

At its core, Kafka scales the simplest of data structures, the log, to the level of distributed systems. The secret to its speed is not an algorithm, but its respect for the physical realities of disks, networks, and parallelism. Every "advanced architecture" ultimately depends on using those physical realities honestly.

Next, we will discuss "Advanced Stream Processing: Choosing Between Flink and Kafka Streams" and examine how stream computing is applied to real business systems.