Load Balancing

One public address - api.example.com resolving to a single IP - can serve millions of requests. No single machine could. The trick is that the address does not point at a server at all; it points at a load balancer, a box whose only job is to spread incoming requests across a pool of identical backend servers behind it. Add more backends and you serve more traffic, all under the same address.

The load balancer sits in the middle

A client opens a connection to the load balancer. The balancer picks one healthy backend, forwards the request, and relays the response. The client never learns which backend answered - it only ever talks to the one address.

Pick an algorithm, then send requests and watch how they land on the pool. Mark a backend down and watch the balancer route around it:

healthy backendmarked downjust received a request

Client

203.0.113.7

→

Load balancer

Round-robin

0 sent

→

srv-1

0 conns

srv-2

0 conns

srv-3

0 conns

Pick an algorithm and send a request to see where it lands.

L4 versus L7 balancing

Where the balancer makes its decision changes what it can see and how fast it is.

	L4 (transport)	L7 (application)
Works on	IP address + port	Full request - URL, headers, cookies
Decision unit	A whole connection	Each individual request
Can route by	Source/destination IP:port only	Path (/images), host, header, cookie
Speed	Very fast - little to inspect	Slower - parses the request
Sees encryption	Passes TLS through untouched	Usually terminates TLS to read the request

An L4 balancer forwards by IP and port without looking inside - cheap and fast, but it only knows "a connection arrived". An L7 balancer reads the application request, so it can send /api to one pool and /static to another, or keep a user's session on one backend.

Balancing algorithms

The balancer needs a rule for choosing a backend. Three common ones, all in the widget above:

• Round-robin - hand each new request to the next backend in order, then wrap around. Simple and even when every backend is equally powerful and every request costs about the same.
• Least-connections - send the request to whichever healthy backend currently has the fewest active connections. Better when requests take very different amounts of time, so a backend stuck on slow work stops getting piled on.
• IP hash / consistent hashing - hash a key (often the client IP) and map it to a backend. The same key always lands on the same backend, which gives a client a stable "home" - useful for caches and session affinity. Consistent hashing is the version that reshuffles as few keys as possible when a backend is added or removed.

info

Least-connections needs the balancer to track live connection counts per backend; round-robin needs only a counter; hashing needs only the key. More awareness costs more state.

#Walking round-robin vs least-connections through the same 10 requests

Take a pool of three backends - A, B, C - where B happens to be handling a few slow, long-running requests already in flight (say 4 active connections, versus 0 on A and C). Send 10 new requests and watch the two algorithms land them differently:

• Round-robin ignores load entirely and just cycles: request 1 to A, 2 to B, 3 to C, 4 to A, 5 to B, 6 to C, 7 to A, 8 to B, 9 to C, 10 to A. After all 10, each backend got roughly 3-4 new requests - but B is now juggling its original 4 slow requests plus its new share, so it ends up more overloaded than A or C even though the balancer split requests "evenly."
• Least-connections checks live counts before each pick. Request 1 sees A:0, B:4, C:0 - ties go to A (or C); say it picks A, now A:1. Request 2 sees A:1, B:4, C:0 - picks C, now C:1. Request 3 sees A:1, B:4, C:1 - a tie again, say A, now A:2. This continues, routing every new request to whichever of A/C is currently lightest, and B gets zero of the first several requests until its existing work finishes and its count drops below the others.

With uniform, short-lived requests the two algorithms perform about the same. The gap only opens up under uneven load - which is precisely the scenario least-connections exists to handle.

Health checks

A pool is only useful if the balancer knows which backends are alive. It periodically probes each one - a TCP connect, or an HTTP request to a /healthz path - and expects a healthy response. A backend that fails the check is pulled out of rotation; requests skip it until it passes again. That is why, in the widget, toggling a backend to "down" makes every algorithm route around it: an unhealthy backend is simply not a candidate.

caution

Health checks must be cheap and independent of the real workload. A check that hits a slow database can mark a healthy server "down" under load and yank it out exactly when you need it - turning a hiccup into an outage.

⚠Sticky sessions and a removed backend do not mix

IP hash / consistent hashing is often used for session affinity ("sticky sessions"): a client's requests all land on the same backend so that in-memory session state (a shopping cart, a logged-in session) stays available without a shared session store. That works fine until the backend a client is stuck to gets pulled out of rotation - deployed over, marked unhealthy, or scaled down. The balancer must now send that client somewhere else, but the new backend has never seen that client's session state, so the user is abruptly logged out, loses their cart, or gets a fresh, empty session with no warning. Plain consistent hashing softens this (only the keys owned by the removed backend move, not all of them), but it does not eliminate it - the backend that owned those keys is still gone, session and all. The durable fix is to not keep session state in the backend's memory at all: put it in a shared store (Redis, a database, or a signed client-side token) so any backend can serve any request regardless of which one handled it last.

Recap

• One address serves millions because it points at a balancer, not a server.
• L4 balances connections by IP:port (fast); L7 balances requests by their content (flexible).
• Round-robin, least-connections, and hashing trade simplicity for awareness.
• Health checks keep dead backends out of the pool automatically.