System Design: Why Message Queues Are Needed and How to Choose One

Consider a common production incident:

A flash sale begins, instantaneous QPS jumps from a few hundred to 50,000, the downstream order service immediately runs out of memory, the database connection pool is exhausted, and the entire request path collapses.

During the postmortem, you will often find that the root cause is not inadequate performance at a single point, but a rate mismatch between upstream and downstream systems. The upstream system sends a sudden burst, the downstream system cannot process it quickly enough, and the entire system is overwhelmed.

A message queue (Message Queue, hereafter MQ) solves this problem. It is essentially the "shock absorber" and "buffer pool" of a distributed system: synchronous calls become asynchronous delivery, and direct connections are replaced by routing through a broker. It may look like only one extra layer, but it changes the system's coupling, throughput model, and fault-tolerance boundaries.

This article explains four things in depth: what problems MQ actually solves, how reliability is guaranteed, how to balance ordering and duplication, and how to choose among mainstream products.

1. What Problems Do Message Queues Solve?

Many people think of MQ as a "first-in, first-out queue." That is correct, but superficial. The real reasons to introduce MQ are the following four engineering problems.

1. Peak Shaving and Load Leveling (Most Common)

This is the most direct value of MQ. The upstream system receives burst traffic from flash sales, notifications, web crawling, or log reporting, while the downstream system has limited processing capacity. Place a queue between them so traffic can wait, allowing the downstream system to consume it at its own pace.

Instantaneous 50k QPS  →  [ Queue (backlog) ]  →  Downstream steady at 2k QPS

The key insight is: MQ does not make processing faster; it prevents traffic from taking your system down. Capacity planning changes from "handle peak load" to "handle average load," reducing costs by an order of magnitude.

2. Decoupling

Synchronous calls require the upstream system to know that the downstream system exists, along with its address, interface, and availability. If any downstream system fails, the upstream system also reports an error.

A classic example is user registration, after which the system must send an SMS, send an email, issue a coupon, add loyalty points, and push the user to the CRM. If all five operations use synchronous RPC, a timeout in any downstream system can slow down or even fail the registration endpoint.

With MQ, the upstream system only publishes a "user registered" event, and every interested system subscribes to it. Adding a downstream system requires no changes to upstream code, and downstream downtime does not affect the primary workflow. The system's extension boundary changes from code coupling to subscription configuration.

3. Asynchronous Processing

Some operations inherently do not require waiting for a result, including analytics reporting, log collection, risk-control records, and push notifications. Performing them synchronously wastes response time and expands the impact of failures.

After making them asynchronous, endpoint response time changes from "the sum of all downstream processing times" to "the time required to deliver a message to MQ," which is usually only a few milliseconds. The user-experience difference might be 800ms versus 50ms.

4. Eventual Consistency (With Transactional Messages)

Strong consistency uses database transactions, while cross-service strong consistency uses 2PC or TCC, both of which are heavyweight and error-prone. Many business scenarios actually require only "eventual consistency": the data simply needs to be processed by the other side eventually.

MQ's "local transaction + message delivery" pattern is one of the standard solutions for distributed eventual consistency. RocketMQ transactional messages and Kafka transactional producers are designed for this scenario.

In short: MQ is not a performance optimization tool; it is a decoupling and buffering layer in the system architecture. It is usually introduced to gain availability, scalability, and resilience.

2. Two Core Models: Point-to-Point vs. Publish/Subscribe

The semantic models of virtually all MQ systems can be classified into these two categories.

Point-to-Point

A message is consumed by only one consumer.

Producer → [Queue] → Consumer A (message deleted after consumption)
                  ↘ Consumer B (does not receive this message)

Typical scenarios include order processing, task distribution, and worker pools. Multiple consumers form a consumer group and compete for messages to increase throughput, not to broadcast them.

Publish/Subscribe

Every subscriber consumes its own copy of a message.

Producer → [Topic] → Subscriber A (receives one copy)
                  → Subscriber B (receives one copy)
                  → Subscriber C (receives one copy)

A typical scenario is broadcasting domain events. For example, when an "order paid" event occurs, the logistics system, loyalty points system, risk-control system, and data warehouse must each process a copy.

Real products combine these two models. Kafka's consumer group mechanism is particularly elegant: consumers within the same group compete for messages (point-to-point), while different groups each consume the complete stream (publish/subscribe). One mechanism supports both semantics.

Before choosing a system, determine whether your downstream systems are "a group of competitors" or "a group of observers." This determines whether you use a Queue or Topic, and a group or fanout.

3. Reliability: Not Losing Messages Is the Baseline

Reliability is where MQ systems most often fail. Losing a single critical message in production, such as a successful payment event, can cause tens of thousands in financial losses. Reliability requires several mechanisms working together; none can be omitted.

1. Persistence

Messages are written to disk rather than stored only in memory. Unconsumed messages must survive a broker crash and restart.

• Kafka: All messages are appended to disk as a log. You can configure the replica count (replication.factor=3) and acknowledgment policy (acks=all).
• RabbitMQ: The queue must be declared durable, and messages must use delivery_mode=2; otherwise, they are lost on restart.
• Redis Streams: Persistence relies on Redis RDB/AOF, but Redis persistence is asynchronous by default. A crash may lose several seconds of data, and you must account for that risk.

Synchronous flushing vs. asynchronous flushing is a critical tradeoff. Synchronous flushing performs worse but is more reliable, while asynchronous flushing performs better but may lose data if the machine loses power. Financial systems choose the former; logging systems choose the latter.

2. Producer Acknowledgments

After a producer delivers a message, the broker must return an acknowledgment. Without one, the producer cannot know whether the message arrived.

• Kafka: acks=0 (consider it successful immediately, fire and forget) / acks=1 (acknowledge once the leader writes it) / acks=all (acknowledge only after all ISR replicas write it).
• RabbitMQ: The publisher confirm mechanism.

For important data, use acks=all + retries + an idempotency key. Do not use acks=0 merely for speed; when something goes wrong, you will have no idea how many messages were lost.

3. Consumer ACKs and Retries

After processing a message, a consumer must explicitly tell the broker, "I have finished processing it." Only then does the broker delete the message. This is called an ACK (acknowledgment).

If a consumer crashes after receiving a message, or throws an exception without sending an ACK, the broker delivers the message again to another consumer. This is called redelivery.

The timing of the ACK matters. Two common modes are:

• ACK after processing (at-least-once, recommended): The message may be consumed more than once, so business logic must be idempotent.
• ACK immediately after receipt (at-most-once): Performance is better, but the message is lost if processing fails.

Most business scenarios should choose the former. ACK is the final safeguard for consumer-side reliability; disabling it leaves the consumer unprotected.

4. Dead Letter Queue (DLQ)

A failed message cannot be retried indefinitely, or a single poison message can block the entire consumption pipeline. The standard approach is: after N failed retries, send it to a DLQ for manual intervention or offline processing.

• RabbitMQ: Natively supports x-dead-letter-exchange.
• Kafka: Has no built-in DLQ. Consumers usually implement one by catching exceptions and writing failed messages to another topic.
• RocketMQ: Provides a dedicated dead-letter topic (%DLQ%consumerGroup).

A DLQ is essential to production stability. Any serious MQ consumer must have a corresponding DLQ policy. A consumer without a DLQ is a production incident waiting to happen.

5. Idempotent Consumption (The Business-Layer Safeguard)

Because redelivery exists, the same message may be consumed multiple times. Consumer logic must be idempotent, or incidents such as duplicate charges, shipments, and SMS messages will occur.

Common ways to implement idempotency include:

• A business-unique key plus a database uniqueness constraint, skipping inserts that conflict.
• Recording processed message IDs in Redis or a database and checking for duplicates before consumption.
• State-machine validation: if an order is already paid, ignore another "payment successful" event.

Idempotency supplements consumption reliability at the business layer. MQ alone cannot guarantee "exactly once," as explained below.

4. Ordering and Duplication: The Enduring Distributed-Systems Tradeoff

Ordering

Global ordering means all messages are consumed strictly in production order. The cost is enormous: you can use only one partition and one consumer, eliminating all parallelism. Do not pursue global ordering except in highly specialized scenarios such as replaying financial events.

Partition ordering means messages with the same key enter the same partition and remain ordered within it. This is the standard engineering solution.

For example, use orderId as the partition key for order events. The same order's "created → paid → shipped → delivered" events are guaranteed to reach the same consumer in order, preserving business ordering, while different orders can be processed in parallel.

Kafka, RocketMQ, and Pulsar all use this model. Choose partition ordering instead of global ordering; it is sufficient for 99% of use cases.

Duplication

MQ delivery semantics fall into three categories:

The industry's open secret is that true end-to-end exactly-once delivery barely exists, or costs so much that it is usually not worthwhile. Kafka's "exactly-once" behavior coordinates transactions among the broker, producer, and consumer, but true exactly-once business processing still requires idempotency in the downstream database. After all that, it is still fundamentally at-least-once plus idempotency.

The practical engineering approach is therefore: use at-least-once by default and implement idempotency in the business layer to simulate exactly-once behavior. This is the most pragmatic and cost-effective solution.

5. How to Choose a Mainstream MQ

The four mainstream options each have their own character. The clearest distinction is what each one does best.

Kafka: The Throughput Leader and First Choice for Stream Processing

• Strengths: Hundreds of thousands of TPS per node, sequential disk writes + zero-copy, the strongest ecosystem with Streams, Connect, and Flink integration, and partition ordering guarantees.
• Weaknesses: Large numbers of topics put pressure on Zookeeper/KRaft metadata; it is unsuitable for complex routing; messages are deleted according to retention policies rather than immediately after consumption.
• Typical scenarios: Log collection, behavioral analytics, real-time data warehouses, event sourcing, CDC data pipelines, and stream processing.
• In short: For "high throughput + ordered streams," Kafka is almost always a sound choice.

RabbitMQ: Flexible Routing for Traditional Business Messages

• Strengths: The AMQP protocol, extremely powerful routing through exchange + routing key + binding, low millisecond-level latency, complete support for priority queues, TTL, DLX, and other features, plus an easy-to-use management interface.
• Weaknesses: Throughput is an order of magnitude lower than Kafka; it handles large backlogs poorly, with performance dropping sharply as messages accumulate; cluster scaling is less natural than with Kafka.
• Typical scenarios: Asynchronous order processing, task distribution, complex routing such as routing different types to different queues, and request/response workflows using RPC over MQ.
• In short: Prefer RabbitMQ for business messages. It has a complete feature set and clear semantics.

Redis Streams: Lightweight and Convenient

• Strengths: Simple deployment, especially if Redis is already in use; consumer groups, ACKs, and persistence are all available; integration cost is low.
• Weaknesses: Reliability boundaries are ambiguous because Redis persistence is asynchronous and may lose several seconds of data in extreme cases; it is unsuitable for large backlogs; operational tooling is less mature than Kafka or RabbitMQ.
• Typical scenarios: Internal asynchronous tasks, user notifications, and lightweight event buses in small and medium-sized projects.
• In short: Do not use it for core transactions, but it is sufficient for asynchronous work where best-effort reliability is acceptable.

RocketMQ / Pulsar: Large Chinese Platforms and Financial Workloads

• RocketMQ: Created by Alibaba and implemented in Java. It has the most complete support for transactional messages, using half messages + transaction status checks, and is particularly well suited to financial and e-commerce order scenarios. Its Chinese documentation and ecosystem are mature.
• Pulsar: Uses a compute-storage separation architecture, provides strong tenant isolation, and supports geo-replication. Its architecture is advanced, but operational complexity is high.

In short: RocketMQ works well for e-commerce and transaction-oriented services in China, while Pulsar is better suited to very large, multi-tenant platform scenarios.

Scenario Selection Table

The selection principle is: choose for the scenario, not from a leaderboard. There is no universally best option, only the most suitable one.

6. Common Pitfalls (Anyone Who Has Encountered Them Will Understand)

Finally, here are the failure modes I see most often. The first three are frequent production incidents.

1. No Alerts for Message Backlogs

When downstream consumption slows or stops, messages accumulate in the queue. By the time you notice, tens of millions may already be waiting and the SLA may already be breached.

Countermeasure: Monitor queue depth (consumer lag) and configure threshold alerts. Evaluate the lower bound of downstream processing capacity, not its average.

2. Lost Messages (No Persistence / Incorrect ACK Handling)

This is the classic basic mistake. A RabbitMQ queue is not declared durable, the producer does not enable confirms, or the consumer acknowledges immediately upon receipt; any one of these oversights can lose messages.

Countermeasure: Core pipelines must use durable + confirm + manual ack + acks=all, and these configuration settings must undergo code review.

3. Non-Idempotent Consumption

Redelivery + non-idempotent processing = duplicate charges or shipments. This causes direct financial loss and is more serious than an outage.

Countermeasure: Design all consumer logic under the assumption that messages may be consumed more than once. A business-unique key + deduplication table should be standard.

4. Single-Partition Performance Bottlenecks

A particular partition key may carry too much data, such as analytics events from one major customer, causing one partition's consumer to fall behind while other partitions remain idle.

Countermeasure: Design partition keys to avoid hotspots, monitor per-partition consumption latency, and repartition when necessary.

5. Using MQ as a Database

Business logic may depend on messages "always being ordered, never being lost, and always being queryable." But MQ is designed for transport, not storage. Messages are removed when their retention period expires.

Countermeasure: Business systems must guarantee their own consistency using databases + state machines. MQ is responsible only for transport.

6. Using Kafka for Everything

Many teams use Kafka regardless of the scenario, even for simple order messages, despite operational costs that are completely disproportionate to the benefits.

Countermeasure: RabbitMQ is often the straightforward choice for small and medium-sized workloads. Before adopting Kafka, ask, "Do I need millions of TPS?"

Summary

The key points are:

1. MQ solves coupling, rate mismatch, and asynchronous processing, not raw performance. Before introducing it, identify which problem it is intended to solve; otherwise, it only adds complexity.
2. Reliability requires several mechanisms working together: persistence + producer ACKs + manual consumer ACKs + retries + DLQ + idempotent consumption. All six are essential. Missing any one of them can cause an incident.
3. Choose partition ordering instead of global ordering. Use at-least-once delivery by default and business-layer idempotency as the most pragmatic engineering approximation of "exactly once."
4. Choose based on the scenario: Kafka for logs and streams, RabbitMQ for business messages and complex routing, RocketMQ for transactional messages, and Redis Streams for lightweight workloads.
5. Pitfalls always hide in the details: backlog monitoring, persistence, and idempotency. Get these three right, and 90% of MQ incidents will never happen.

One final takeaway:

A message queue is not a silver bullet. It trades complexity for resilience. Avoid it when you do not need it; when you do, use it professionally. Reliability is never merely a configuration option; it is a design philosophy.

Next, we will discuss "System Design: Rate Limiting and Circuit Breaking" and examine how to control traffic once it arrives.