Load Balancing Strategies: Round Robin / Least Connections / Consistent Hashing

A single 8-core, 32 GB application server starts becoming unstable at 5,000 QPS. CPU usage surges to 90%, GC pauses grow longer, and P99 latency jumps from 50 ms to 800 ms. You add memory, tune JVM parameters, and optimize SQL, but the marginal gains keep shrinking. At this point, the real problem is not single-server performance: the single-server model itself has reached its limit.

The solution is straightforward: add more machines. Three servers handling 2,000 QPS each are far more stable than one server struggling with 6,000 QPS. You also gain fault tolerance: if one goes down, the other two can keep serving traffic.

But adding machines immediately creates a new problem: when a user request arrives, which machine should receive it? This is what a load balancer (LB) solves. It does much more than simply "distribute requests evenly." Choose the wrong strategy, and some machines will be overwhelmed while others sit idle; cache hit rates will collapse; users will lose their login sessions; or adding capacity will suddenly remap traffic and bring down the cluster. This article explains the key load-balancing decisions in one place.

1. Why Load Balancing Is Essential

A load balancer adds a layer between clients and backend services. It performs three tasks:

1. Distribution: Routes requests across multiple backend instances according to a strategy, enabling horizontal scaling.
2. Health management: Continuously probes backends and automatically removes failed instances without affecting users.
3. Consolidation: Exposes only one entry point externally, such as a VIP or domain name, making backend scaling transparent to clients.

Without an LB, a "multi-server" deployment is merely a collection of isolated machines. An LB turns it into an operationally manageable cluster. Nearly every modern architecture, from a single Nginx instance to a global GSLB, is built on this abstraction.

2. L4 vs. L7: Split at the TCP Layer or the HTTP Layer?

Load balancers fall into two major categories based on the layer at which they operate. This is the most fundamental selection decision.

L4 (transport-layer load balancing): Examines only the IP address and port, then forwards TCP connections. It does not inspect or parse HTTP content. Examples include HAProxy in tcp mode, LVS, cloud-provider NLBs, and IPVS.

• Advantages: Extremely fast. It does little more than rewrite packet headers and forward traffic, so throughput can reach hundreds of thousands of PPS. It is protocol-independent and can forward MySQL, Redis, and gRPC traffic.
• Disadvantages: It cannot see URLs, headers, or cookies, so it cannot route by path or perform HTTP-based canary releases.

L7 (application-layer load balancing): Terminates TCP, reads the HTTP request, and makes routing decisions based on the URL, headers, cookies, or even the request body. Examples include Nginx, HAProxy in http mode, cloud-provider ALBs, and Envoy.

• Advantages: Flexible. /api/* can go to a Java cluster, /static/* can go to a CDN origin, and requests with X-Canary: true can be routed to a new version. It can also rewrite requests, compress responses, enforce rate limits, and provide WAF capabilities.
• Disadvantages: Heavyweight. Every connection requires HTTP parsing, so latency and CPU overhead are higher than with L4. It is also tied to specific protocols, primarily HTTP/HTTPS.

A practical rule: use L7 at the edge, where you need HTTPS termination, path routing, and canary releases; use L4 for internal service-to-service communication, where throughput and protocol independence matter. This is the default layering in most medium-sized and large systems.

3. Five Major Algorithms: Who Gets the Request?

The routing algorithm is the core of an LB strategy. Choose correctly, and cluster utilization remains even with low latency. Choose poorly, and you will either overload individual machines or destroy cache effectiveness.

1. Round Robin: A→B→C→A→B→C, assigning requests in sequence. It is simple and even over time, making it suitable when all backend machines have the same configuration and each request has a similar processing cost. The drawback is that if request costs vary greatly, such as a 1 ms cache lookup versus a 500 ms complex SQL query, some machines may randomly accumulate many slow requests.

2. Weighted Round Robin: Assigns each machine a weight and distributes requests in proportion to those weights. An 8-core machine might have a weight of 2, while a 4-core machine has a weight of 1, so the 8-core machine receives twice as many requests. This works well when machine configurations differ, such as mixed instance types or old and new hardware. It is also useful for canary releases, where the new version's weight is gradually increased from 1 to 100. Nginx uses smooth weighted round robin (Smooth WRR) by default, spreading weighted assignments more evenly over time instead of creating bursts.

3. Least Connections: Sends the request to the backend with the fewest currently active connections. This is the most "intelligent" simple algorithm because it naturally handles uneven request costs. A machine processing many slow requests has more active connections, so new requests automatically avoid it. It is suitable when request processing times vary widely, such as API gateways, persistent connections, WebSockets, and file uploads. The tradeoff is that the LB must maintain connection counts for every backend, making it stateful, but this is not a problem for modern load balancers. HAProxy uses leastconn by default.

4. IP Hash: Hashes the client IP address and maps it to a fixed backend. The same client always reaches the same machine. Its primary purpose is session persistence, discussed in Section 5: if a session is stored in local memory, the same user must return to the same machine. The drawback is that the hash distribution may be uneven. A large NAT egress IP can represent enormous traffic and overwhelm one machine. When nodes are added or removed, mappings for nearly all users change, invalidating every session.

5. Consistent Hashing: Maps both backends and requests onto a ring, with each request moving clockwise to the nearest node. Its key property is that when nodes are added or removed, only requests in adjacent ranges migrate, while all others remain unchanged. The migration volume is approximately 1/N, where N is the number of nodes, far better than the near-total remapping caused by ordinary hashing. It is suitable for caching layers, such as Redis Cluster, Memcached, and CDN origins, because cache invalidation sends requests through to the database. It is commonly combined with "virtual nodes" to produce a more even distribution.

Algorithm comparison:

A practical rule: start with least connections by default because it adapts best to real traffic. Use IP hash or consistent hashing only when you have a clear need for session persistence or cache affinity.

4. Health Checks: Failed Backends Must Be Removed Automatically

Another core responsibility of an LB is to send traffic only to healthy backends. Without health checks, failed machines continue receiving requests. The LB treats timeouts as ordinary distribution outcomes, and users experience what looks like "half of the cluster being down."

There are two modes:

Passive health checks: The LB does not probe proactively. It observes the results of real requests instead. After N consecutive failures, it marks the backend as down and stops sending it traffic, then retries after a period. The advantage is zero additional probe traffic. The disadvantage is that user requests act as probes, so the first requests during recovery will fail. HAProxy and Nginx support this by default.

Active health checks: The LB periodically sends probes to the backend, such as HTTP GET /health requests or TCP connections, and determines health from the response. This detects failures earlier and does not depend on real traffic. The open-source edition of Nginx provides only a simplified version, while Nginx Plus, HAProxy, ALB, and Envoy provide full active health checking. The tradeoff is that the backend must provide a lightweight, side-effect-free health endpoint, and probes must not become a burden themselves. Do not query the database inside /health.

Production practice: use both. Passive checks provide second-level failure isolation, while active checks provide periodic confirmation and recovery. Health endpoints should be layered: separate liveness, meaning the process is alive, from readiness, meaning dependencies are ready and the instance can receive traffic. If the database is disconnected, readiness should return 503 so the LB temporarily removes the instance.

5. Session Persistence: Route the Same User to the Same Machine

Many services remain stateful: sessions may live in local memory, files may be cached locally, and WebSocket connections are bound to a particular process. These scenarios require the same user, or the same connection, to always reach the same backend. This is called a sticky session, or session persistence.

Implementation options:

• IP hash: As described above, this is coarse-grained. The problem is that thousands of users behind a large NAT gateway share one IP address and may all be directed to the same machine.
• Cookie stickiness: The LB sets a cookie in the first response, such as SERVERID=node3. Subsequent requests carrying that cookie return to node3. HAProxy, Nginx, and ALB all support this. Its fine granularity makes it the mainstream approach.
• Headers or business fields: Apply consistent hashing to a user ID or tenant ID. This is suitable for multi-tenant SaaS systems.

The cost is clear: if a backend fails, users pinned to it lose their sessions, unless sessions are stored externally in Redis. Session persistence is therefore a "necessary evil." A better architecture moves state elsewhere, using Redis or JWTs, so backends remain stateless and any machine can handle any request. That is what enables truly flexible horizontal scaling. If stateless operation is possible, avoid sticky sessions.

6. Tool Selection: Nginx / ALB / HAProxy / Envoy

Selection guideline: for low operational overhead in the cloud → ALB; for control and cost efficiency → self-managed Nginx; for maximum performance or L4 → HAProxy; for a service mesh → Envoy. Large companies often combine several layers rather than choosing only one.

7. Multi-Layer Balancing: Large Architectures Are "Onions"

A truly highly available system uses more than one load-balancing layer. Traffic converges layer by layer from the outside in:

• Global layer (GSLB / DNS): Resolves www.example.com to the nearest region based on geography and carrier. A user in Beijing receives the VIP for the Beijing data center, while a user in Tokyo receives the Tokyo endpoint. This is implemented with Anycast or GeoDNS.
• Regional layer (L7 ALB / Nginx): Serves as the data-center entry point, terminates HTTPS, routes URLs to different services, such as /order to the order cluster and /search to the search cluster, and performs header-based canary releases.
• Service layer (L4 / sidecar): Distributes connections or requests among instances within a service. It usually uses NLB/IPVS at L4 or an Envoy sidecar in a service mesh for fine-grained L7 control.

Each layer has its own responsibility. The global layer handles regional failures, such as an entire data center going offline. The regional layer handles business routing and canary releases. The service layer handles instance-level load balancing and health isolation. No single layer can solve every problem independently. A lone L7 ALB cannot handle cross-region failover, while DNS alone cannot perform HTTP-based canary routing. This onion-like layering is how large systems address the reality that any layer can fail.

Summary

1. A load balancer solves three problems: traffic distribution, health isolation, and entry-point consolidation. Without it, a "multi-server" deployment is just a collection of isolated machines.
2. L4 sees IP addresses and ports and is fast but limited; L7 understands HTTP and is flexible but heavier. Using L7 at the edge and L4 internally is the default layering for most systems.
3. Choose least connections by default because it adapts best to real traffic with uneven request durations. Use consistent hashing for caching layers and IP hash or cookies for session persistence.
4. Combine passive and active health checks. Separate liveness from readiness, and do not query the database in a probe.
5. Session persistence is a necessary evil. The best solution is to externalize state and keep backends stateless. Large systems use multi-layer balancing, GSLB → L7 → L4, to converge traffic layer by layer.

Key takeaway: Load balancing is not about choosing the single strongest algorithm. It is about letting each layer do what it does best: the edge understands the business, internal layers understand throughput, and the global layer understands geography. Putting each decision at the right layer matters more than pursuing perfection at any one layer.