RabbitMQ vs Kafka vs Redis Streams: Choosing a Message Queue

Whenever "add a message queue" comes up in an architecture review, an argument almost inevitably follows: Kafka or RabbitMQ? In recent years, another option has joined the discussion: Redis Streams. By the end, it often becomes clear that the people arguing have not distinguished the fact that these three systems are fundamentally different things. They simply happen to share the label "message queue."

The purpose of this article is not to take sides. All three are outstanding tools in their respective domains. The key question is: is your problem actually the kind of problem each one excels at solving? We will first explain their core models, then discuss how to choose based on your requirements.

1. First Understand This: Message Queues Are Not All the Same Thing

Many people conflate these concepts, but if you read only one section, read this one:

These three statements determine every difference that follows:

• RabbitMQ is a message dispatcher. A message enters the system, is routed to a queue, is taken and acknowledged by a consumer, and then disappears. Its core purpose is to "reliably deliver this message to its intended recipient."
• Kafka is a log. Producers append records to the end of the log, while each consumer maintains an offset indicating how far it has read. Consumers can replay records, rewind, or read independently of one another. A message does not disappear simply because it has been read.
• Redis Streams is a lightweight stream between the other two. Its data model resembles Kafka, with an append-only log and consumer groups, but it runs in Redis memory and does not have Kafka's heavyweight ecosystem.

Once you understand this distinction, none of the design choices that follow, including routing, ACKs, partitions, and retention policies, are arbitrary. They all arise from these three underlying models.

2. RabbitMQ: Classic AMQP and the Art of Routing

2.1 Core Model: Exchange + Queue

The essence of RabbitMQ is decoupling routing logic. Producers do not place messages directly into queues. They publish them to an Exchange, which sends each message to one or more Queues according to a set of rules. This rule-based step is RabbitMQ's strongest capability.

Remember these four Exchange types and you will have what you need:

• direct: Performs an exact match on routing_key. For example, with routing_key = "error", only queues bound to error receive the message.
• topic: Performs pattern matching and supports wildcards in routing_key. logs.*.error matches both logs.app.error and logs.db.error. This is the most commonly used type.
• fanout: Broadcasts the message unchanged to every bound queue and ignores routing_key. It is suitable when "multiple systems must process the same event."
• header: Matches message headers instead of relying on routing_key. It is uncommon in real projects, but can occasionally be invaluable for complex routing conditions.

Consider a typical scenario: an order system publishes an order.created event that must be consumed by inventory, loyalty, notification, and risk-control services. In RabbitMQ, you create one topic exchange, give each service its own queue, and bind each queue to a key such as order.# or order.created as needed. They do not interfere with one another, and each acknowledges and retries messages independently.

2.2 ACKs, Retries, and Dead-Letter Queues: The Moat Around Reliable Delivery

RabbitMQ takes the requirement that "every message must be processed correctly" very seriously:

• Manual ACK: A consumer sends basic_ack only after it has finished processing a message. If the process crashes during handling, no ACK is sent, so the message is redelivered to another consumer.
• prefetch: Limits the number of unacknowledged messages assigned to a single consumer, preventing one consumer from being overwhelmed while others remain idle.
• Dead-letter queue (DLX): Messages can be routed to a designated dead-letter exchange when they receive basic_reject / basic_nack without requeueing, when their TTL expires, or when the queue is full. Failed messages are handled centrally instead of being lost.
• Delayed queue: Traditionally implemented with TTL + DLX. RabbitMQ now also has the official rabbitmq_delayed_message_exchange plugin, which provides a more direct approach.

These mechanisms make RabbitMQ almost unrivaled for scenarios such as task distribution, order processing, and payment callbacks, where every message must be preserved and produce a result.

2.3 Where RabbitMQ Fits

• Task queues: Background jobs such as Worker pools, Celery, and Sidekiq
• Event notifications: Fanout of business events to multiple subscribers
• RPC: Asynchronous RPC using a reply-to queue + correlation_id
• Complex routing: Distributing multiple message types to different consumers according to rules
• Traditional microservice communication: Reliable delivery, ACKs, and retries

Poor fit: Messages that must be consumed repeatedly through replay, extremely high-throughput log streams at the million-TPS level, or long-term event sourcing.

2.4 Deployment and Operations

• Written in Erlang, with distribution based on clustering + mirrored queues. Quorum Queue is now recommended. It is based on Raft and is more reliable than the old mirror queue.
• A single node can handle tens of thousands to more than one hundred thousand TPS, with further reductions across clusters.
• Operational complexity is moderate, and both the CLI (rabbitmqctl) and management UI are mature.

3. Kafka: A Distributed Commit Log Built for High Throughput

3.1 Core Model: Topic → Partition → Offset

Kafka's core abstraction is not a queue. It is a partitioned log:

• A Topic is divided into N Partitions.
• Each Partition is an ordered, immutable, continuously appended log.
• Messages are ordered within a Partition but not across Partitions.
• Every message has a monotonically increasing offset.
• Consumers maintain their own offsets and can read from any position. If they want to rewind and reread data, they can simply do so.

The key consequence of this model is that consumption is completely decoupled from production-side "deletion." Producers only append to the end of the log. Consumers maintain their own offsets without affecting one another. The same message can be consumed once by each of N different consumer groups, with none of them needing to know that the others exist.

3.2 Consumer Groups: Scaling and Parallelism

The Consumer Group is central to Kafka:

• Consumers in the same Group divide all Partitions among themselves. A Partition is consumed by only one consumer within the Group.
• Need more consumption parallelism? Add Partitions and consumers. However, when the number of consumers exceeds the number of Partitions, the extra consumers do no work.
• Different Groups do not affect one another. Each independently consumes the complete dataset.

This mechanism makes Kafka a natural fit for "one event stream consumed independently by N downstream systems," which is a core requirement of event-driven architectures and data platforms.

3.3 Message Retention: Kafka's Real Killer Feature

RabbitMQ deletes messages after consumption, while Kafka retains them for seven days by default, with configurable retention, regardless of whether consumers have read them.

This distinction looks simple, but it completely changes what you can do:

• Event replay: Launch a new consumer, start from an offset one week in the past, and feed the entire historical dataset through it for recomputation. RabbitMQ cannot do this.
• Failure recovery: If a consumer recovers after being down for three hours, it resumes from its last committed offset without losing data.
• Event Sourcing: Store every state change as an event so the current state can be reconstructed at any time.
• Stream processing: Use Kafka + Streams / Flink / Spark Streaming to treat messages as streams and compute windowed aggregations and Joins.
• Data pipelines (ETL): Use Kafka Connect to send database binlogs, logs, and metrics into Kafka, where downstream systems can consume them as needed.

3.4 Performance and Reliability

• Sequential disk writes + zero-copy (sendfile) allow a single node to handle hundreds of thousands to more than one million TPS with ease.
• Replicas + ISR (In-Sync Replicas) provide durability.
• An idempotent producer + transactions can provide exactly-once semantics, at a performance cost.
• Ordering is guaranteed only within a Partition. Global ordering requires a single Partition, which limits throughput. This is an inherent Kafka trade-off.

3.5 Where Kafka Fits

• Log aggregation: Application logs, access logs, and audit logs
• Event streams / stream processing: Real-time data warehouses, user behavior analytics, and risk control
• Data pipelines (ETL): CDC and data synchronization
• Event Sourcing / CQRS: Workloads that require replay or state reconstruction
• High-throughput message bus: The event backbone between microservices

Poor fit: Complex routing rules, since Kafka has nothing like an exchange and relies on Topic naming conventions; task queues that require strict reliable delivery, retries, and dead-letter handling for every message, which are cumbersome to implement; and small projects where the operational cost is not justified.

3.6 Deployment and Operations

• Heavyweight. Zookeeper / KRaft, Brokers, replication, monitoring with Burrow and Kafka Exporter, Schema Registry, Connect, and the rest of the stack can occupy an entire team.
• KRaft mode, which removes Zookeeper, has simplified deployment considerably, but Kafka remains enterprise-grade infrastructure.
• Managed Kafka services in the cloud, such as MSK, Confluent Cloud, and Upstash, can eliminate much of the operational burden.

4. Redis Streams: A Lightweight Stream Hidden Inside Redis

4.1 What It Is

The Streams data structure introduced in Redis 5.0 is essentially an in-memory append-only log + consumer group model. Its API closely resembles Kafka:

• XADD writes a message and automatically generates an ID in <timestamp>-<seq> format.
• XREAD / XREADGROUP consume messages.
• XGROUP CREATE creates a consumer group.
• XPENDING / XCLAIM handle orphaned messages by allowing unacknowledged messages from a failed consumer to be claimed by another.
• XACK acknowledges a message.
• XLEN, XRANGE, and XREVRANGE support random reads and replay.
• MAXLEN / MINID limit stream length and perform Trim operations.

See the pattern? This is a miniature Kafka, with the same log + consumer group model, except that it runs in Redis memory.

4.2 Key Differences from Kafka

4.3 Key Differences from RabbitMQ

Redis Streams has no exchange / routing. If you need to distribute messages to different queues according to rules, RabbitMQ is the better choice. Routing in Redis Streams can essentially be simulated only by manually using different stream names.

4.4 Where Redis Streams Fits

• Event streams for small and medium-sized projects: User presence, IM messages, and real-time counters
• Cross-process Pub/Sub + message retention: Native Redis Pub/Sub loses messages when a subscriber disconnects; Streams solves this problem
• Lightweight task queues: If you do not want to deploy RabbitMQ, Streams + consumer groups can handle the job
• Redis-heavy workloads such as cache warming and real-time leaderboards: Reuse the existing Redis deployment and reduce operational overhead

Poor fit: Large datasets or event streams requiring long retention, because they may not fit in memory; workloads with extremely strict durability requirements where absolutely no data can be lost, because Redis uses asynchronous persistence by default and a crash can lose the most recent few seconds; and complex routing rules.

4.5 An Often-Overlooked Point: Streams ≠ Redis Pub/Sub

The original Redis Pub/Sub model delivers messages only while subscribers are connected. Disconnected subscribers miss messages, and the messages are not retained. Streams was introduced largely to address this limitation. If your project still uses Pub/Sub to distribute critical events, it is time to switch to Streams.

5. Side-by-Side Comparison: The Differences in One Table

6. Selection Decisions: Choose by Requirements, Not Popularity

Everything above can be condensed into a decision tree:

1. Do you need "task distribution," "reliable delivery," or "complex routing"?

Examples include order processing, email delivery, payment callbacks, Worker pools, cross-service RPC, and routing one event to different consumers according to rules.

→ Choose RabbitMQ. Its exchanges, ACKs, DLX, and prefetch are built for these scenarios. Kafka can also do this, but it is awkward because you must implement retries, dead-letter handling, and delayed queues yourself.

2. Do you need a "high-throughput event stream," "logs," or "replay"?

Examples include user behavior logs, monitoring metrics, risk-control event streams, real-time data warehouses, CDC data pipelines, event sourcing, and multiple downstream systems independently consuming complete copies of the data.

→ Choose Kafka. This is its irreplaceable domain. Replay, multiple consumer groups, million-level TPS, and long-term retention are capabilities that RabbitMQ / Redis Streams cannot provide at the same level.

3. Do you need a "lightweight event stream," already use Redis, or want to avoid introducing new middleware?

Examples include IM messages, user presence, real-time notifications for small and medium-sized applications, and Web applications that rely heavily on Redis for caching.

→ Choose Redis Streams. One data structure handles the message stream, with almost no additional operational cost.

4. Still unsure? Then answer one question first: do you need to replay messages?

• Need replay → Choose between Kafka and Redis Streams based on scale and existing infrastructure
• No replay needed, and every message should disappear after processing → RabbitMQ

7. Do Not Adopt Kafka Just to "Use Kafka"

This deserves its own section because I have seen too many teams fall into this trap.

Kafka is so prominent in the technology industry that it often becomes a form of "political correctness." As soon as teams adopt microservices or an event-driven architecture, their first reaction is "use Kafka." But Kafka's costs are very real:

• Operational cost: Operating a Kafka cluster, including Zookeeper / KRaft, monitoring, Connect, and Schema Registry, carries a burden comparable to adding half a middleware team.
• Complexity: How many Partitions should you create? How many replicas? How long should data be retained? How should consumer rebalances be handled? How should offsets be committed? You must understand all of these questions, or production incidents are inevitable.
• Poor value at low throughput: If your workload is only a few thousand TPS, one RabbitMQ node is enough. A Kafka cluster is overengineering.

I once saw a team force order status changes, a classic task-distribution workload where every message required reliable delivery, retries, and dead-letter handling, into Kafka. To implement "retry after consumption failure + dead-letter handling," they went through an elaborate retry topic + dead letter topic design. The code was extremely verbose and bugs were constant. With RabbitMQ, the same work would have taken half a day.

I have also seen the reverse: a log aggregation workload handling terabytes of data per day ran on RabbitMQ for six months. The cluster was unstable every day, but after the team migrated to Kafka, everything ran smoothly.

Both failures came from the same mistake: choosing a tool for its "reputation" instead of choosing it for the actual "problem."

So here is the final point:

When choosing a message queue, first understand the problem you need to solve. Use RabbitMQ for task distribution, Kafka for high-throughput event streams, and Redis Streams for lightweight workloads. Do not let a tool's reputation make architectural decisions for you.

Summary

1. The three systems are fundamentally different: RabbitMQ uses "queues + exchanges" for message distribution and deletes messages after consumption. Kafka uses a "distributed commit log" for retained, replayable event streams. Redis Streams provides a "lightweight in-memory stream," essentially a miniature Kafka. These differences exist at the design-philosophy level, not the configuration level.

2. The central selection question is whether you need replay: If you need replay and multiple consumer groups that each consume the complete stream, choose Kafka or Redis Streams. If you do not need replay but require reliable delivery and complex routing, choose RabbitMQ.

3. RabbitMQ's moat is routing and reliable delivery: Four exchange types, ACKs, dead-letter handling, and prefetch. It remains unmatched for task queues, RPC, and event notifications.

4. Kafka's moat is high throughput + replay: Million-level TPS, time-based retention, multiple consumer groups, and Partition-level parallelism. It is irreplaceable for logs, stream processing, ETL, and event sourcing.

5. Redis Streams is the "good enough and easy to operate" choice: If you already have Redis, the workload is not large, and you need a lightweight event stream, choose it and avoid adding another middleware system.

Key takeaway: choose a message queue based on the shape of your problem, not on which technology looks best on a resume. Kafka is not an upgraded version of RabbitMQ; they are different things. Redis Streams is a third option, not a cheap substitute for RabbitMQ.