Redis's Role in Distributed Systems

I have seen far too many teams treat Redis as an "expiring HashMap." Their resumes say they are "experts in Redis," but ask how to prevent a distributed lock from being released by the wrong owner, why rate limiting uses a sorted set instead of INCR, or when to choose Streams over pub/sub, and they cannot answer. Redis is standard infrastructure in distributed systems not merely because its cache is fast, but because it provides frequently needed primitives for locks, rate limiting, messaging, leaderboards, and sessions in one system. This article is not about caching. It explains Redis's genuinely irreplaceable roles in distributed systems and the reliability requirements of each one.

1. What Role Does Redis Actually Play in a Distributed System?

Classified by the problems they solve, Redis's responsibilities combine five distinct roles:

1. Coordination: distributed locks that determine which instance may execute a task.
2. Admission: rate limiting that controls request rates per user, per IP, or globally.
3. Communication: pub/sub broadcasts and persistent messaging with Streams.
4. State: session storage shared across multiple application instances.
5. Ranking: real-time top-N leaderboards.

Without Redis, you might need to deploy ZooKeeper, Kafka, a dedicated session service, and a separate ranking service to cover these problems. Redis provides all five in one place, but each role has its own reliability requirements: locks must prevent incorrect release, rate limiting must be accurate, Stream messages must be acknowledged, and sessions need an appropriate persistence policy. Let us examine them one by one.

2. Distributed Locks: SET NX EX + Unique Token + Lua Release

Distributed locks are among the easiest primitives to implement incorrectly. The broken version usually looks like this:

SET lock:order 1 NX EX 30     # Acquire the lock
# ... business logic ...
DEL lock:order                # Release the lock

This code will eventually fail in production. The problem is that DEL does not verify the owner. Suppose A acquires the lock but stalls for more than 30 seconds. The lock expires automatically, B acquires it and begins working, and then A finally resumes and runs DEL. A has now deleted B's lock, allowing C to acquire it as well. The lock has completely failed.

The correct implementation has three steps. Store a unique token when acquiring the lock:

SET lock:order <token> NX EX 30

Generate <token> with a UUID or randomBytes; it is the unique proof of ownership for this acquisition. When releasing the lock, use a Lua script to atomically compare the token and delete the key:

if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('DEL', KEYS[1])
else
    return 0
end

Why must this use Lua? A separate GET check followed by DEL consists of two operations with a race between them. Lua scripts execute atomically inside Redis, so no other command can interleave between the comparison and deletion. This is the non-negotiable requirement for a distributed lock. Without it, the lock is merely decorative.

How should you choose the expiration time? Set it too short, and the lock expires before the work finishes. Set it too long, and everyone else waits unnecessarily after the owner crashes. In production, do two things together: set the expiration slightly above the operation's P99 duration, such as a 30-second lock for work that normally takes 5 seconds, and add a background watchdog that extends the TTL before expiration while the process still owns the lock. Redisson's watchdog uses this pattern, and implementing it yourself is not particularly complex.

Should you use Redlock? A lock stored in one Redis instance can be lost during primary-replica failover. If the primary fails before replicating the lock, another client may acquire it on the promoted replica. Redlock is the multi-node design proposed by the Redis author: acquire the lock independently on N Redis nodes and consider it successful only when a majority succeed. Martin Kleppmann published a well-known critique whose central argument is that Redlock relies on sufficiently consistent node clocks. During GC pauses, process suspension, or network partitions, a client may believe it still holds a lock that has already expired. My practical view is that a single-instance lock with tolerance for rare failures is sufficient for most applications. For genuinely strong consistency, such as financial deductions, do not use a Redis lock. Use a database row lock or etcd. A Redis lock fundamentally provides high performance and best-effort coordination, not a correctness guarantee.

3. Rate Limiting: A Sorted-Set Sliding Window Is More Accurate Than INCR

The most naive rate limiter uses INCR + EXPIRE:

INCR  rate:user:123
EXPIRE rate:user:123 60    # Apply a 60-second expiration when the window starts

This is a fixed-window algorithm, and its problem is obvious: it can admit twice the intended traffic at a window boundary. Send all 100 allowed requests at second 59, then another 100 at second 01 of the next window, and 200 requests pass within 1.1 seconds.

A sliding window implemented with a sorted set is much more accurate. Store each request as a member whose score is its timestamp:

ZADD             rate:user:123  <now_ms>  <req_id>
ZREMRANGEBYSCORE rate:user:123  0  <now_ms - 60000>   # Remove requests older than 60 seconds
ZCARD            rate:user:123                          # Count requests in the current window
EXPIRE           rate:user:123  70                      # Do not forget this, or cold keys never expire

Check whether the ZCARD result exceeds the threshold. These three operations must be wrapped in a Lua script and executed atomically, or concurrent requests can exceed the limit. Per-user, per-IP, and global limits differ only in key granularity: rate:user:123, rate:ip:1.2.3.4, or rate:global.

Accuracy requirements: obtain timestamps from Redis's server-side TIME, not the client clock, which may drift. Every member must be unique, using a request ID or nanoseconds plus a random value. Otherwise, a sorted set deduplicates requests that share a timestamp, causing undercounting. Sorted sets consume substantial memory on extremely high-frequency endpoints, so an approximate sliding window based on segmented counters may be a better performance tradeoff.

4. Messaging: Pub/Sub Broadcasts vs. Persistent Streams

Redis provides two messaging models, and choosing the wrong one can cause an incident.

Pub/sub is broadcast-only and has no persistence:

PUBLISH   chan "hello"     # Publish
SUBSCRIBE chan             # Subscribe

Its behavior is simple: messages are lost when subscribers are offline; if no subscriber is present at publish time, the message disappears immediately; there are no consumer groups, acknowledgments, or replay. It is suitable only when subscribers are guaranteed to be online and message loss is acceptable. One example is broadcasting cache invalidations between application instances: one instance deletes an entry and tells the others to delete it too. Treating pub/sub as Kafka is one of the most common ways to get this wrong.

Streams provide a persistent queue with consumer groups, similar to Kafka:

XADD        orders '*'  id 123 amount 50          # Produce; * lets Redis generate an ordered ID
XGROUP      CREATE  orders g1 0                   # Create a consumer group
XREADGROUP  GROUP g1 c1 COUNT 10 STREAMS orders > # Consume within the group; > means undelivered messages
XACK        orders g1 <msg_id>                    # Acknowledge consumption
XPENDING    orders g1                             # Inspect unacknowledged messages
XAUTOCLAIM  orders g1 c2 60000 0                  # Transfer timed-out messages to another consumer

A Stream's reliability depends on acknowledgments. After XREADGROUP returns a message, the message enters the PEL, or pending entries list. It is not considered fully consumed until XACK succeeds. If a consumer crashes, its messages remain in the PEL and another consumer can take them over with XAUTOCLAIM. Reading without acknowledging causes an ever-growing message backlog. This is the most common operational failure with Streams. Also limit stream length with a command such as XADD ... MAXLEN ~ 10000; otherwise, an unbounded message history eventually consumes all available Redis memory.

5. Leaderboards: The Natural Domain of Sorted Sets

Leaderboards are one use case where competing tools struggle to match Redis. A sorted set is inherently ordered by score, and both real-time updates and range queries run in O(logN):

ZADD            leaderboard 1500 alice 1200 bob 1800 carol  # Update scores
ZREVRANGE       leaderboard 0 9  WITHSCORES                  # Top 10 in descending order
ZREVRANGEBYSCORE leaderboard 1500 1000                       # Members within a score range
ZREVRANK        leaderboard alice                            # Alice's rank
ZINCRBY         leaderboard 50 alice                         # Add points in real time

Why not use MySQL with ORDER BY score LIMIT 10? At scale, sorting the table for every query becomes disastrous. The sorted set's skip-list structure makes top-N retrieval close to constant overhead. For live-stream donation rankings, game ladders, and trending-topic scores, sorted sets are almost always the best choice whenever the workload combines real-time ordering with frequent updates.

6. Session Storage: TTLs for Distributed Sessions

A monolithic application can keep sessions in memory. Once it scales horizontally to multiple instances, sessions must move to shared external storage. Otherwise, a user who logs in through instance A may have the next request routed to instance B and appear to be logged out. Redis is the de facto standard for distributed sessions:

SET  session:<sid>  <user_data_json>  EX 86400    # Expire after 24 hours
GET  session:<sid>

The reliability requirement is simple: lost session data means users are logged out. A Redis deployment that stores sessions must therefore enable persistence with AOF or RDB. Do not store sessions in a purely ephemeral Redis instance. Use sliding expiration for active users by renewing the TTL with EXPIRE on each access, while inactive sessions expire naturally. Keys should be traceable, such as session:<sid>, so administrators can revoke a session immediately with DEL session:<sid> when an account is banned.

7. Do Not Forget Redis's Own High Availability

The easiest fact to overlook is that a single Redis instance is a single point of failure. Every design above, including locks, rate limiting, messaging, and sessions, assumes Redis remains available. If Redis fails, the entire distributed system loses its shared memory. Redis itself therefore needs high availability:

• Replication: one primary with multiple replicas and optional read scaling. Failover is manual, so production deployments should not stop at this layer.
• Sentinel: monitoring added to primary-replica replication, with automatic primary election after a failure. Suitable for small and medium-sized deployments.
• Cluster: data is sharded across multiple groups, each with its own primary and replicas. It provides horizontal scaling and automatic failover and is the standard choice at large scale.

My practical recommendation is to keep sessions, locks, and other data that must not be lost away from cache-only instances. Store them in Redis with AOF persistence and Sentinel or Cluster protection. Rate limits, leaderboards, and other state that can be reconstructed may use ordinary cache instances and tolerate brief inaccuracies during failover. Separating important data from disposable cache data across different Redis deployments provides the best balance of cost and reliability.

Summary

1. Redis is a unified provider of five distributed-system primitives: coordination, admission, communication, state, and ranking. It is not merely an expiring HashMap, and each role has distinct reliability requirements.
2. Safe distributed locks require Lua-based release: store a unique token when acquiring the lock, then use Lua to atomically compare that token and run DEL. Without this step, the wrong owner can release the lock. Do not use Redis locks for strong-consistency workloads.
3. A sorted-set sliding window is more accurate than INCR for rate limiting: fixed windows can admit twice the intended traffic at their boundaries. Atomic Lua execution, the Redis server clock, and unique members are the three requirements for accuracy.
4. Pub/sub is an ephemeral online broadcast mechanism; Streams provide a reliable Kafka-like queue. XACK is essential. Reading without acknowledging causes messages to accumulate.
5. A single Redis instance is a single point of failure. Sessions, locks, and other state that must not be lost require persistence plus Sentinel or Cluster protection.

In one sentence: Redis's performance often makes its reliability boundaries easy to forget, but incorrectly released locks, exceeded rate limits, lost messages, and vanished sessions all cause real financial damage. Use its speed, but always understand how each primitive can fail.