Redis Cluster: Sharding, High Availability, and Failover

I have broken Redis Cluster down several times, and every time someone asks the same question: "Our primary-replica setup with Sentinel is running perfectly well. Why should we use Cluster?" Most of the time, the answer is that you should not. But when a single machine runs out of memory, single-core write throughput is saturated, or a full primary-replica synchronization involves tens of gigabytes and takes forever, Cluster stops being a question of whether to use it and becomes a question of how to deploy it reliably.

This article explains Redis Cluster's sharding model, replication and failover, client routing, cross-slot restrictions, and the decision boundary between Cluster and Sentinel. The configuration examples target production use, and the pitfalls are called out explicitly.

1. The Ceiling of a Single Redis Instance: When Cluster Becomes Necessary

First, let us clarify when you should not use Cluster, because that is the more common case. If a Redis process is running well, forcing it into Cluster mode only adds operational complexity: clients must support the cluster protocol, cross-slot operations are restricted, some batch commands become unavailable, and migrations require careful handling. Until you hit one of the following two walls, Cluster is overengineering.

Wall 1: memory. Redis is an in-memory system, so dataset size is approximately equal to memory consumption. A single machine has a physical memory limit. On a server with 256GB of RAM, once one Redis instance holds more than 100GB, latency from fork operations during RDB generation or AOF rewriting and doubled copy-on-write memory usage start to hurt. This is covered in detail in Redis Persistence. Pushing farther turns the instance into a time bomb: recovery after a restart takes tens of minutes, fork pauses last seconds, and the process may be killed by the OOM killer. Once the dataset exceeds the comfortable range for one machine, typically 30-50GB, sharding is the only way forward.

Wall 2: single-core throughput. Redis uses a single-threaded main execution model. Since Redis 6.0, network I/O can use multiple threads, but command execution is still single-threaded. The write QPS ceiling of one instance is roughly 100,000-150,000, with somewhat higher read throughput. When an instance continuously reaches this level, CPU usage is saturated and latency becomes unstable. Adding replicas can offload reads, but writes still have to run on one core of the primary. The only way to scale write throughput at that point is horizontal sharding.

Beyond these two walls, there is one implicit requirement: reducing the failure blast radius. If a single instance goes down, the entire keyspace becomes unavailable. After sharding, one failed shard affects only its portion of the keyspace until a replica fails over. This is a secondary benefit, not the main reason to adopt Cluster. The primary reason is always that one machine can no longer hold the data or one core can no longer handle the writes.

Single Redis instance (including primary-replica + Sentinel)

         ┌────────────────────────────┐
 Client →│   Master (full dataset)    │ ← Single-core write ceiling: ~100K-150K QPS
         │   Memory limit ≈ host RAM  │ ← fork latency, doubled COW memory
         └────────────────────────────┘
                    │ replication
               ┌────┴────┐
               ▼         ▼
            Replica   Replica        ← Offload reads only; writes remain constrained

Redis Cluster solves this by dividing the keyspace into 16,384 slots, distributing those slots across multiple primary nodes, and attaching replicas to each primary for high availability. Write throughput scales with the number of shards, available memory grows with the number of nodes, and a failure in one shard does not affect the others.

Redis Cluster (3 primaries, 3 replicas, distributed slots)

     ┌──────────────┐   ┌──────────────┐   ┌────────────────┐
     │  Master A    │   │  Master B    │   │  Master C      │
     │ Slots 0–5460 │   │ Slots 5461–10922│ │ Slots 10923–16383│
     └──────┬───────┘   └──────┬───────┘   └──────┬─────────┘
            │ replication      │                  │
            ▼                  ▼                  ▼
       ┌────────┐         ┌────────┐         ┌────────┐
       │ReplicaA│         │ReplicaB│         │ReplicaC│
       └────────┘         └────────┘         └────────┘

   Write throughput = 3 × one core   Total memory = 3 × one host   ← Horizontal scaling

2. The Sharding Model: 16,384 Slots and CRC16

Redis Cluster's sharding model can be summarized in one line: slot = CRC16(key) % 16384. Understand this line, and you understand how Cluster distributes data.

Several details matter:

1. Why CRC16 instead of a key hash or consistent hashing? Redis Cluster does not use consistent hashing. It uses a fixed number of slots plus hash modulo. Consistent hashing minimizes data movement when nodes are added or removed, but it can produce uneven distribution and makes range queries difficult. Redis instead divides the keyspace into 16,384 fixed slots. Nodes own slot ranges, and scaling operations migrate whole slots rather than individual placement rules. This gives migration, routing, and redirection clear boundaries.

2. Why 16,384 rather than 65,536 or more? Redis author Salvatore Sanfilippo answered this specifically on the mailing list. There are three reasons:

• Gossip bandwidth. Cluster nodes exchange state through the Gossip protocol. Each message carries a bitmap describing the slots owned by the sender. 16,384 bits equals 2KB. With 65,536 slots, the bitmap would be 8KB, making every Gossip message four times larger and consuming substantial bandwidth as the cluster grows.
• Practical node limits. A real Cluster has far fewer than 16,384 nodes. The author considered deployments above 1,000 nodes unrealistic, so additional slots would mostly be wasted.
• Slot compression. The bitmap can be compressed further with run-length encoding, and 16,384 slots provide effective compression at a practical scale.

3. Is the whole key hashed, or only part of it? By default, Redis hashes the complete key. The exception is a hash tag, which the next section covers.

key="user:1001:profile"
        │
        ▼
   CRC16("user:1001:profile")        ← 16-bit CRC
        │
        ▼
   mod 16384                         ← Produces a value from 0 to 16383
        │
        ▼
   slot = 12543
        │
        ▼
   Look up slot→node mapping → Master C

4. How does a node know who owns each slot? Every node maintains a global slot → node mapping, stored in nodes.conf and synchronized between nodes through Gossip. Clients also cache this mapping. Before sending a command, a client calculates the slot locally, consults the mapping, and sends the command directly to the target node instead of relying on redirection for every request. This is why cluster-aware clients such as Lettuce, go-redis, ioredis, and redis-py-cluster matter more than redis-cli -c: the former cache the mapping and perform client-side routing, while the latter relies on server-side redirection for each request.

A Cluster with three primary nodes distributes slots almost evenly by default:

$ redis-cli -p 7000 cluster nodes
<id-A> 127.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460
<id-B> 127.0.0.1:7001@17001 master      - 0 0 2 connected 5461-10922
<id-C> 127.0.0.1:7002@17002 master      - 0 0 3 connected 10923-16383

Notice that each node exposes two ports: 7000 is the client port, while @17000, the client port plus 10,000, is the cluster bus port. Nodes use a binary protocol over the cluster bus for Gossip, failure detection, and configuration synchronization. Both ports must be allowed through production security groups and firewalls. Otherwise, nodes cannot detect one another and the cluster cannot form.

3. Hash Tags: Keeping Related Keys in the Same Slot

Cluster shards by key hash, which has an immediate consequence: an operation involving multiple keys cannot run if those keys hash to different slots. For example:

MSET user:1:name alice  user:2:name bob
(error) CROSSSLOT Keys in request don't hash to the same slot

user:1:name and user:2:name hash to different slots, so Cluster rejects the command. This is the cross-slot restriction, covered in detail later. In many applications, however, a group of keys is logically related and must be operated on together: a user's shopping cart, fields belonging to one order, or multiple keys changed by a transaction. Hash tags exist for exactly this purpose.

The rule is simple: if a key contains {...}, Redis hashes only the substring inside the braces.

key                 hashed content  destination
────────────────────────────────────────────────────
user:{1001}:cart    "1001"          slot(CRC16("1001"))
user:{1001}:profile "1001"          slot(CRC16("1001"))   ← Same slot!
user:{1001}:orders  "1001"          slot(CRC16("1001"))   ← Same slot!
order:{5002}:items  "5002"          slot(CRC16("5002"))   ← Different slot

As long as the tag inside the braces is identical, all three keys are guaranteed to land in the same slot and on the same node. This makes the following command valid:

MSET user:{1001}:name alice  user:{1001}:email [email protected]
OK                              ← Same slot, allowed

Transactions, Lua scripts, and MGET can all operate across multiple keys as long as those keys use the same tag.

Rules for using hash tags:

1. Do not overuse them. Forcing many hot keys into one tag binds all of them to one slot and one node. That node becomes a hotspot, defeating the purpose of sharding. Use tags only for groups of keys that must be updated atomically for business reasons, such as keys associated with one user or order.
2. Tags must have enough cardinality. If the tag is {2024}, every item from 2024 lands in one slot and creates immediate skew. Tags should distribute distinct business entities, so use entity IDs such as {userId} or {orderId} rather than categories.
3. The rule uses the first { through the first }. user:{1001}:{extra} uses only 1001. With nested braces such as foo{bar{baz}, Redis uses the first outer pair. Empty braces {} have no effect, and Redis hashes the entire key normally.

In one sentence: a hash tag is Cluster's escape hatch for saying, "These keys must stay together." Used correctly, it enables atomic cross-key operations. Used incorrectly, it concentrates enough traffic to overwhelm one node.

4. Primary-Replica Replication and Gossip: Cluster's High-Availability Foundation

Sharding alone is not enough. If the primary for a shard goes down, every slot assigned to that shard becomes unavailable. Cluster therefore attaches replicas to every primary so that a replica can take over when the primary fails. This layer is fundamentally the same as ordinary Redis replication, but Cluster adds several important details.

Every slot has one primary and at least one replica. Three primaries and three replicas are the minimum viable configuration:

Shard A: Master A (slots 0-5460)       ← replication → Replica A
Shard B: Master B (slots 5461-10922)   ← replication → Replica B
Shard C: Master C (slots 10923-16383)  ← replication → Replica C

A primary handles reads and writes for its slots. By default, replicas only replicate and do not serve external traffic unless cluster-allow-reads-when-down is enabled or read requests are explicitly sent to a replica. Replication works like ordinary primary-replica Redis: full synchronization through RDB plus an incremental buffer, followed by command propagation. That mechanism is not expanded here.

The key Cluster-specific question is not replication itself, but how nodes know whether their peers are still alive. The answer is Gossip.

At short intervals, each node randomly selects several peers and sends PING messages. The interval is derived from cluster-node-timeout. Each PING includes:

• The sender's state, including whether it is alive and whether it is a master or replica
• The slots it owns, represented by the slot bitmap
• Summaries of other nodes known to the sender, including their IP addresses and states, in the gossip section

The peer responds with a PONG carrying the same categories of information. Cluster state continuously propagates and converges through these exchanges. When a node fails, the entire cluster generally learns about it through Gossip within tens of seconds.

cluster-node-timeout is the most important tuning parameter, with a default of 15 seconds:

• Set it too low, and a brief network disturbance can cause false failure detection and unnecessary failovers.
• Set it too high, and genuine failures take too long to detect, extending the outage window.
• A practical production range is 5-10 seconds on a stable internal network and 15-30 seconds across data centers or less reliable networks. Evaluate it together with the heartbeat interval, cluster-ping-interval, and make sure heartbeats occur much more frequently than the timeout to avoid false positives during transient disruptions.

Do not forget these settings:

port 7000                        # Client port
cluster-enabled yes              # Enable cluster mode; without this, it is ordinary Redis
cluster-config-file nodes-7000.conf  # Cluster state file maintained by Redis; do not edit manually
cluster-node-timeout 15000       # Time without contact before PFAIL; use 5-15s in production
cluster-announce-ip 10.0.0.1     # Explicitly advertise the reachable IP behind NAT or containers
cluster-announce-port 7000
cluster-announce-bus-port 17000
cluster-require-full-coverage yes # Reject writes cluster-wide if any slot has no owner; default yes

cluster-announce-* is critical in container deployments. In Docker or Kubernetes, Redis sees its own container-private IP, but Gossip broadcasts that address to other nodes, which may be unable to reach it. You must use cluster-announce-ip to advertise the host IP or another externally reachable address. Otherwise, the cluster may fail to form inside containers or its nodes may be unable to connect to one another.

5. Automatic Failover: Detection, Election, and Promotion

When a primary fails, Cluster must automatically promote one of its replicas to a new primary and transfer ownership of the slots. This failover process has three stages: detection → election → promotion.

Stage 1: detection (PFAIL → FAIL).

• Node A sends a PING to node B. If no PONG arrives within cluster-node-timeout, A marks B as PFAIL, meaning probable failure.
• A broadcasts through Gossip that B may have failed.
• When a majority of primary nodes have marked B as PFAIL, the cluster upgrades B's state to FAIL, meaning confirmed failure.
• The FAIL state then propagates across the cluster through Gossip.

Why require confirmation from a majority of primaries instead of trusting one node? To prevent false positives. A node might be suffering from its own network problem and unable to see B even though B is healthy. If a majority of primaries cannot see B, an actual failure is much more likely.

Stage 2: election, initiated by replicas and decided by primary votes.

• B's replica, called B' here, notices that B is in FAIL state and starts an election.
• B' increments its config epoch, a monotonically increasing configuration version, and requests votes from the primaries.
• Each primary follows a first-request-wins rule and casts one vote per election epoch.
• B' wins after receiving votes from a majority of primary nodes.

This majority-vote mechanism follows the same quorum principle as Raft and Paxos: only one replica can be elected at a time, preventing two replicas from both believing they are the new primary and causing split-brain behavior.

The election also uses ordered preferences to avoid promoting a stale replica:

• replica-priority, formerly slave-priority: lower numbers receive higher priority. A value of 0 means the replica must never be promoted, which is useful for a backup-only node.
• Replication offset: when priorities are equal, the replica with the highest offset, and therefore the newest data, is preferred.
• Run ID: if all other values are equal, the lexicographically smallest run ID wins.

Stage 3: promotion (B' → new primary).

• After winning the election, B' promotes itself to master.
• It broadcasts a PONG through Gossip containing the new config epoch and the slots it now owns.
• Other nodes update their slot maps and point B's slots to B'.
• Client requests for those slots are subsequently routed to B'.

The entire failover usually completes within tens of seconds, consisting of cluster-node-timeout plus a short election delay. During this window, writes targeting slots in the failed shard will fail. Reads may still be possible if replica reads are enabled. This is Cluster's failure window and one reason critical data requires persistence. The new primary starts from the replicated data it already has, so writes from the last few seconds may be lost if they were not replicated before the failure.

Master B fails                       Majority of primaries confirms via Gossip → FAIL
      │                                                    │
      ▼                                                    ▼
  Replica B' sees B FAIL → starts election → primaries vote (quorum)
                                                           │
                                                           ▼
                                       B' wins → promoted to master
                                       → broadcasts PONG (new epoch)
                                       → cluster maps B's slots to B'

Manual failover. Sometimes you want to switch proactively, for example before taking a primary offline for maintenance, rather than waiting for it to fail. Run CLUSTER FAILOVER on the replica:

# Run on the replica to take over proactively
CLUSTER FAILOVER          # Default TAKEOVER; wait for full synchronization before switching
CLUSTER FAILOVER FORCE    # Force the switch without waiting for primary confirmation
CLUSTER FAILOVER TAKEOVER # Seize control manually without quorum; split-brain risk, use carefully

For normal operations, use CLUSTER FAILOVER with no arguments. It lets the replica catch up with the primary before switching and is the safest option. TAKEOVER is for disaster recovery and should not be used routinely.

6. Client Routing: MOVED and ASK

A Cluster client's central responsibility is to calculate the slot locally → look up the slot→node mapping → send the request directly to the target node. The mapping is not static, however. Nodes are added or removed, slots migrate, and primaries fail over. When a client contacts the wrong node, Redis returns one of two redirections. These responses are the heart of the Cluster protocol.

MOVED: permanent redirection.

The client sends a command to A, but the slot now belongs to C and A is no longer responsible for it. A returns:

-MOVED 12543 127.0.0.1:7002

This means that slot 12543 permanently belongs to 127.0.0.1:7002. The client should:

1. Update its local slot→node mapping so that slot 12543 points to port 7002.
2. Retry the command against port 7002.

MOVED signals that the mapping is stale. After updating the map, subsequent requests for the same slot go directly to the correct node without another redirection. Cluster-aware clients refresh the complete map, or update it incrementally, after receiving MOVED so that their routing information remains current.

ASK: temporary redirection during migration.

While a slot is migrating from A to C, it temporarily exists on both nodes. Some keys remain on A while others have already moved to C. When the client asks A for a key:

• If the key remains on A, A handles the request normally.
• If the key has moved to C, A returns -ASK 12543 127.0.0.1:7002.

ASK means, "Ask C for this request only, but do not update the mapping." The client should:

1. Leave its local mapping unchanged. This redirection applies only during migration. Although C will ultimately own the slot, updating the map prematurely can cause incorrect routing while migration is still in progress.
2. Send an ASKING command to C, followed by the actual command. ASKING acts as a one-time pass that tells C, "I know this slot is being imported; process this request." Without it, C rejects the command because the slot does not formally belong to C yet.
3. Redirect only this request. The next request for the same slot still follows the local mapping to A until migration completes and a MOVED response updates the mapping.

The essential difference between MOVED and ASK:

Client sends GET user:{1001}:profile
        │
  Local mapping: slot(CRC16("1001")) → A
        │
        ▼
  Send to A ──┬── key is on A → normal response
              │
              └── key moved to C → -ASK 12543 C:7002
                                      │
                                      ▼
                       Client: ASKING → GET
                       (send to C without updating local mapping)

Practical considerations:

• Use a cluster-aware client. Let the client library maintain mappings and handle MOVED and ASK. Do not implement redirection logic yourself. go-redis, ioredis, Lettuce, and redis-py are mature options.
• Monitor MOVED churn. Frequent MOVED responses in production indicate that the client's mapping is badly out of date. This usually happens immediately after failover or resharding when the client cache has not refreshed promptly. In a healthy steady-state cluster, MOVED responses should be close to zero.
• redis-cli -c is only for demonstrations. It follows redirections automatically, but it relies on server-side redirection each time and does not cache the mapping. Its performance is poor, so do not use it as a production client.

7. Cross-Slot Restrictions: Why MGET, Transactions, and Lua Fail

Sharding makes multi-key operations something that must be handled carefully. The rule is simple: for any command involving multiple keys, every key must map to the same slot, or Redis returns a CROSSSLOT error.

Affected operations include:

1. Multi-key commands. MGET, MSET, DEL key1 key2, and EXISTS k1 k2 work only when all keys share a slot:

MGET user:{1001}:name user:{1001}:email      ← Same tag and slot, OK
MGET user:1001 user:1002                      ← Different slots, CROSSSLOT

2. Transactions (MULTI/EXEC). Every key in a transaction must share the same slot:

MULTI
HSET user:{1001}:profile name alice
SADD user:{1001}:tags vip
EXEC                              ← Keys share a tag, OK

For cross-slot keys, Redis reports the error during the MULTI phase.

3. Lua scripts. Every key accessed by a script must be declared explicitly through KEYS[] before execution, and all of those keys must share a slot:

EVAL "return redis.call('GET', KEYS[1])" 1 user:{1001}:name
                                   ↑
                     Must be declared through KEYS[];
                     do not construct keys inside the script

Why? Cluster must determine before execution which keys the script will access and whether they share a slot. It cannot perform that validation when keys are not declared through KEYS[], because Cluster does not parse the script body to discover them. Therefore, Lua scripts running under Cluster must never construct key names dynamically. Every accessed key must be passed as an argument and use the same hash tag.

4. Set operations. Multi-key set operations such as SUNION, SINTER, SDIFF, and ZUNIONSTORE also require all keys to share one slot.

There is only one workaround: hash tags. Bind keys that must be operated on together to the same slot with {tag}. This is an intentional escape hatch in the Cluster design: Redis recognizes that some keys belong together at the application level and provides an explicit mechanism for declaring that relationship.

A real design tradeoff: suppose a user system stores profile, cart, and order data in separate groups of keys. You want a multi-key operation that retrieves all information for one user.

• Use {userId} as the tag: user:{1001}:profile, user:{1001}:cart. All keys for that user share one slot, enabling MGET and transactions. The cost is that all data for that user is bound to one node, so an extremely active user can become a hotspot.
• Do not use a tag and distribute the keys independently. Multi-key operations become impossible, requiring multiple single-key requests. The cost is losing atomicity and batch efficiency.

The right choice depends on the application. In most cases, tagging by entity ID is appropriate. A single entity's data is unlikely to become large enough to overwhelm a slot, while losing atomic operations forces more complex coordination into the application layer.

8. Cluster vs. Sentinel: Which One Should You Choose?

This is the most frequently asked question. Start by separating their responsibilities:

In one sentence: Sentinel provides high availability for a single dataset; Cluster provides sharding plus high availability.

Sentinel solves the problem, "My Redis deployment must not be a single point of failure." It adds several Sentinel processes, usually an odd number such as three or five, to a primary-replica setup. They monitor the primary, promote a replica automatically when it fails, and notify clients to switch to the new primary. The data is not sharded. Every key is still stored on one primary, so write throughput remains limited by one core and memory by one machine.

Cluster solves the problem, "The dataset no longer fits on one machine, or one core cannot handle the writes." It adds sharding on top of high availability. Each shard contains its own primary-replica setup and failover mechanism, while data is distributed across shards.

Decision tree:

Can your Redis dataset, including expected growth, fit comfortably in one machine's memory
(typically ≤ 30-50GB)?
│
├── Yes → Has write QPS reached the single-core ceiling (~100K-150K)?
│         │
│         ├── No  → Sentinel, or primary-replica. Do not add Cluster complexity.
│         └── Yes → Cluster. Scaling write throughput requires sharding.
│
└── No → Cluster. The data volume requires sharding, so there is no alternative.

My practical view:

• Sentinel is sufficient for most small and medium-sized systems. One primary-replica setup plus three Sentinels can support sessions, caching, rate limiting, and leaderboards when the dataset is within a few dozen gigabytes and write QPS is in the tens of thousands. This covers 90% of use cases. Sentinel clients are simpler, multi-key operations have no restrictions, and the operational burden is lower.
• Use Cluster only when the dataset is genuinely large or write throughput has truly saturated one core. Do not adopt Cluster merely to appear more advanced. Its client complexity, cross-slot restrictions, and migration procedures impose ongoing costs on the team.
• A hybrid strategy is also common: separate Redis instances by business domain. Use one Sentinel deployment for caching, another for sessions, and another for counters. Each remains independent, manageable in scale, and isolated from the others. This is often easier to operate than placing everything in one large Cluster.

Whether to use Cluster depends on whether the dataset fits on one machine and whether one core can handle the writes, not on whether the architecture looks distributed enough. Sentinel is high availability; Cluster is sharding plus high availability. When you do not need sharding, Cluster only creates unnecessary work.

9. Scaling in Practice: Slot Migration and Split-Brain Recovery

Cluster operations are truly tested by scaling and split-brain incidents. Deploying Cluster without experience in either area is a gamble.

Scaling Out: Add Nodes and Migrate Slots

To add a new primary D, first join it to the cluster and then assign some slots to it:

# 1. Start a Redis instance with cluster-enabled yes
# 2. Add it to the cluster
redis-cli --cluster add-node 127.0.0.1:7003 127.0.0.1:7000

# 3. Move some slots from existing nodes to D
#    This is interactive and asks how many slots to move and where to send them
redis-cli --cluster reshard 127.0.0.1:7000
# How many slots? -> 4096
# Receiving node id? -> <D's node id>
# Source node ids? -> all (or specify sources)

# 4. Add a replica for D
redis-cli --cluster add-node 127.0.0.1:7004 127.0.0.1:7000 \
          --cluster-slave --cluster-master-id <D's node id>

At the protocol level, reshard breaks slot migration into transfers of individual keys. Migrating one slot works as follows:

# On source node A, mark slot 1234 as migrating to D
CLUSTER SETSLOT 1234 MIGRATING <D-node-id>

# On destination node D, mark slot 1234 as importing from A
CLUSTER SETSLOT 1234 IMPORTING <A-node-id>

# Move keys individually or in batches
MIGRATE <D-ip> <D-port> "" 0 5000 KEYS key1 key2 ...

# After all keys move, point slot 1234 to D on every node
CLUSTER SETSLOT 1234 NODE <D-node-id>

During migration, the slot is in a dual-node state: A still contains keys that have not moved, while D contains keys already transferred. This is exactly when ASK redirections occur. If a client asks A for a key already moved to D, A returns ASK and directs the client to D. Migration is transparent to the application, although some requests incur one additional ASK hop.

Practical considerations:

• Resharding is online, but do not run it during peak traffic. Migrating a large key, such as a list, hash, or sorted set with tens of thousands of fields, blocks both the source and destination nodes. Run redis-cli --bigkeys first to identify large keys, then migrate during a quiet period.
• Use --cluster-yes to skip confirmation only in scripts. During manual operations, read every interactive reshard prompt carefully.
• Run --cluster check after migration to verify that slots are distributed evenly.

Scaling In: Remove a Node and Reclaim Its Slots

Scaling in reverses the expansion process: migrate D's slots to other nodes, then remove D from the cluster.

# 1. Move D's slots to other nodes
#    Run reshard with D as the source and another node as the destination
redis-cli --cluster reshard 127.0.0.1:7000
# How many slots? -> <number of slots currently owned by D>
# Receiving node? -> <id of a retained node>
# Source? -> <D's node id>

# 2. Remove nodes, deleting the replica before the primary
redis-cli --cluster del-node 127.0.0.1:7000 <D-from-node-id>
redis-cli --cluster del-node 127.0.0.1:7000 <D-node-id>

The most common mistake is deleting D before migrating all of its slots away. Those slots are then left without an owner. With cluster-require-full-coverage yes, the entire cluster immediately rejects writes. Before deleting a primary, confirm that it owns zero slots.

Split Brain: Two Primaries Both Believe They Own the Same Slot

Split brain is Cluster's most dangerous failure mode. A typical case occurs when a network partition divides the cluster into two groups:

• Partition 1: A, the primary for slots 0-5460, plus a majority of primary nodes
• Partition 2: A's replica A' plus a minority of primary nodes

If partition 2 incorrectly promotes A', even though it should not be able to win without a majority of primary votes, both A and A' may believe they are the primary for slots 0-5460. Both accept writes, and their data diverges.

Redis Cluster uses several mechanisms to prevent split brain:

1. Election quorum. A replica must receive votes from a majority of primary nodes before it can be promoted. Partition 2 contains only a minority of primaries, so A' should not be elected. This is the central defense.

2. Monotonically increasing, globally unique config epochs. Every successful failover assigns the new primary a config epoch higher than those of existing primaries. When Gossip state converges, the higher epoch supersedes the lower one. Even if two primaries briefly coexist, the one with the lower epoch loses ownership after the network recovers.

3. min-replicas-to-write and min-replicas-max-lag. These settings require a primary to see at least N replicas whose lag remains within X seconds before accepting writes. An isolated primary that cannot see a replica rejects writes, preventing it from continuing to accept divergent updates.

min-replicas-to-write 1     # Require at least one visible replica before accepting writes; default 0, recommend 1
min-replicas-max-lag 10     # Replica lag must not exceed 10 seconds

Practical recommendations:

• min-replicas-to-write 1 should be standard in production. Even with only one replica, it prevents an isolated primary from continuing to accept writes. The tradeoff is that the primary temporarily rejects writes if its replica fails, but that is far easier to handle than divergent data after split brain.
• Use an odd number of primary nodes. Three, five, or seven primaries allow a quorum to form. With an even number such as four or six, an equal network split leaves neither side with a majority and makes the cluster unwritable.
• Keep cluster-require-full-coverage yes, the default. If any slot has no owner, the entire cluster rejects writes. It is better to stop completely than to accept writes against only part of the dataset and face a painful merge later.
• What if split brain has already occurred? Do not attempt an automatic merge. Stop one side, compare the data manually, reset the incorrect nodes with CLUSTER RESET, and join them again. Split-brain recovery is difficult; prevention is always easier.

10. Operations Checklist

Review this list before launch to prevent most Cluster incidents:

1. Node count: at least three primaries and three replicas. Three primaries provide quorum, and replicas enable per-shard failover.
2. Set cluster-node-timeout to 5-15 seconds. The default 15 seconds can be slow; 5-10 seconds shortens the failure window on a stable internal network.
3. Configure cluster-announce-* in container or NAT environments. Otherwise, Gossip advertises private addresses that other nodes cannot reach.
4. Allow the cluster bus port, the client port plus 10,000, through the firewall. Gossip uses it, and the cluster cannot form without it.
5. Set min-replicas-to-write 1. This is the lowest-cost defense against split brain.
6. Use a client library that supports Cluster. Do not use redis-cli -c as a production client, and do not use an old library without Cluster support.
7. Identify large keys before migration. Run redis-cli --bigkeys and reshard during off-peak hours to prevent migration from stalling application traffic.
8. Enable persistence. Every shard primary should use AOF with everysec and hybrid mode. Cluster manages high availability, not durability. After failover, the new primary has only the data it replicated; anything not replicated is lost.
9. Inspect CLUSTER NODES and CLUSTER INFO regularly. Verify that cluster_state is ok, slots are distributed evenly, and no node remains in fail state for an extended period.
10. Practice failover. Deliberately stop a primary and verify that its replica takes over within the expected time. High availability that has never been tested is not high availability.

Summary

1. Cluster fundamentally provides sharding plus high availability: 16,384 slots are assigned to multiple primaries using CRC16, and every primary has replicas for automatic failover. The only hard reasons to adopt Cluster are that the dataset no longer fits on one machine or one core can no longer handle the writes. Sentinel is enough otherwise.
2. slot = CRC16(key) % 16384 is the foundation of everything. A hash tag, {tag}, is the escape hatch for keeping related keys in one slot. Used correctly, it enables atomic cross-key operations; used incorrectly, it creates hotspots.
3. MOVED is a permanent redirection that updates the mapping, while ASK is a temporary migration redirection that uses ASKING without updating the mapping. Cluster-aware clients handle both for you; do not implement the protocol yourself.
4. Cross-slot behavior is a hard Cluster restriction. Every key used by MGET, transactions, Lua scripts, or set operations must share a slot, or Redis returns CROSSSLOT. The only solution is a hash tag.
5. Cluster is not Sentinel. Sentinel provides high availability for one dataset; Cluster provides sharding plus high availability. Choose based on dataset size and write throughput, not on whether the architecture looks distributed enough.
6. Split brain is prevented by three defenses: quorum, config epochs, and min-replicas-to-write. An odd number of primaries and minimum-replica write protection should be standard.

Sharding is not a silver bullet. It replaces the problem of fitting data on one machine with an entire new set of problems: coordinating multiple machines, migrating data, and preventing split brain. Cluster's value is not merely that it works, but that you understand what happens behind every MOVED, ASK, and failover, as well as when it can fail.

Further reading:

• Redis Persistence: RDB vs. AOF and Hybrid Mode — Cluster provides high availability, but each shard still needs AOF for durability.
• Redis's Role in Distributed Systems — Considerations for distributed locks, rate limiting, Streams, and leaderboards under Cluster.