The Request Path - From DNS to Database, with a Latency and Availability Budget
- Every large system is the same pipeline: DNS, traffic steering, CDN edge, load balancer, proxy or gateway, service, cache, database. Each box exists to remove one bottleneck or one single point of failure (SPOF).
- Latency adds, availability multiplies. A request that crosses six tiers pays the sum of their times and survives only if all of them do. Three serial 99.9% components give 99.7%, about 26 hours of downtime a year instead of 8.76.
- Redundancy is the only way up. Two independent 99% replicas in parallel give 1 - 0.012 = 99.99%, but only if their failures are really independent (different hosts, different racks, different AZs).
- Tail latency is a fan-out problem. If one call in 100 is slow, a request that fans out to 100 calls is slow 63% of the time. Watch p99, not the mean (The Tail at Scale).
- To find the slow hop, measure each one:
curl -wtimings, then tracing, then per-service rate, errors and duration.
This page is the server-side map for the whole module. The browser-side story (what the client does when you type a URL) is told once in The URL journey; here we follow the same request after it leaves the client and lands in a data centre. Each hop gets one line and a link to its owner page, and this page keeps the shared vocabulary and the arithmetic.
Mental model
An airport on a busy day. Signage and the booking system tell you which airport and terminal to use (DNS and steering). A regional hub close to home takes you most of the way (CDN edge). Security lanes spread the crowd over many scanners (load balancer). A gate agent checks your papers and sends you to the right plane (proxy or gateway). The plane is the actual work (service), the baggage hall is the quick-access pile (cache), and the records office behind it holds the truth (database).
Where it breaks: an airport does not retry a lost passenger on its own, and it does not run ten identical copies of every counter. Networks do both: a client or proxy will quietly resend a request that timed out, and every tier is cloned many times. Cloning and retrying are what create the failure modes this page is about.
The path is a chain of caches and queues. Each tier either answers from local state (a cache hit) or forwards to the next tier and pays its latency. A request's total time is the sum of the stages it visits, and its chance of success is the product of each stage's success rate, exactly like composing functions that can each throw.
Where it breaks: function calls are in one process and fail together; these stages fail independently, which is both the cause of the multiplication and the reason parallel replicas help. Also, in-process calls do not have a network timeout, a retry policy or a queue in the middle.
How it works
One request, eight hops
Take a user opening an API-backed page on a service that handles 1 million requests per second. The request touches these tiers in order. The "what it removes" column is the point of each box: if you cannot say which bottleneck or SPOF a box removes, you probably do not need it.
| Hop | What it does | What it removes | Owner page |
|---|---|---|---|
| DNS | Turns the name into an address; the answer can depend on who asks | Hard-coded addresses; one fixed location | DNS |
| Traffic steering (GSLB, anycast) | Picks a region or PoP by health, load and proximity | One region as a SPOF; a far-away default | Global traffic steering |
| CDN edge | Terminates TCP and TLS near the user, answers cacheable requests | Long round trips; origin load | CDN |
| Load balancer (VIP) | Spreads connections or requests over a pool of healthy backends | One server as a SPOF; one server as the capacity limit | Load balancing, Consistent hashing |
| Proxy or gateway | Routes by path or host, authenticates, limits, retries | Every service re-implementing the same edge logic | Proxies and gateways, Rate limiting |
| Service | Runs the business logic; calls other services | Nothing alone; it is the work | Timeouts, retries, backoff |
| Cache | Keeps hot answers in memory beside the service | Repeated database reads | HTTP caching |
| Database | Holds the durable truth | Losing data on a restart | your database's documentation |
Two cross-cutting notes. TLS is terminated at the CDN edge or the load balancer in most designs; see TLS for the handshake and for re-encrypting to the origin. And inside a cloud account, which subnet and route table each hop lives in is the subject of VPC subnets and routing. For flood defence see Network attacks.
Where the time goes: a latency budget
A latency budget divides a target (here 200 ms at the 99th percentile) over the hops so that each team knows its share. The allocations below are a design exercise for the walkthrough, not measurements:
| Stage | Allocation | Why this share |
|---|---|---|
| DNS lookup | 20 ms | a cold lookup walks root, TLD and the zone's name servers; most users hit a resolver cache and pay far less (DNS) |
| TCP + TLS handshakes | 60 ms | two round trips before the first request byte (TCP one, TLS 1.3 one) to a nearby PoP; 2 x 30 ms RTT = 60 ms (TLS) |
| CDN or edge processing | 10 ms | cache lookup, request routing, TLS record handling |
| Load balancer | 1 ms | a forwarding decision, mostly in memory |
| Service | 40 ms | business logic and downstream calls |
| Database | 20 ms | one indexed read that missed the cache |
| Slack | 49 ms | 200 minus the sum 151; absorbs queueing and bad luck |
Two honest caveats. First, percentiles do not add: the p99 of a sum is not the sum of the p99s. Adding p99 allocations is a conservative planning rule (every stage at its tail at once is rarer than any one stage at its tail), which is why teams keep slack. Second, the 60 ms handshake line only holds because the PoP is close. Light in optical fibre travels at about 200,000 km/s, so a round trip to a far origin has a hard floor from distance alone:
| Path | Distance (great circle, approximate) | RTT floor = 2 x distance / 200,000 km/s |
|---|---|---|
| Same data centre | under 1 km | negligible; real RTT is about 0.5 ms from switching and the stack |
| Cross-AZ, same region | tens of km | real RTT about 1 to 2 ms |
| Virginia to Ireland | about 5,500 km | 55 ms |
| Virginia to Sydney | about 15,700 km | 157 ms |
Real round trips sit above the floor because cables do not follow great circles and every router queues; use your provider's published inter-region numbers, and see Latency and capacity numbers for the table to memorise. The 157 ms Sydney figure is derived step by step on the CDN page, along with why three round trips make a far origin cost 471 ms at the floor.
Where the availability goes: a product
Availability is the fraction of time (or requests) a component works. The nines are shorthand. Downtime per year is (1 - A) x 8,760 hours; per month use 720 hours (30 days).
| Availability | Downtime per year | Downtime per 30-day month |
|---|---|---|
| 99% (two nines) | 87.6 hours | 7.2 hours |
| 99.9% (three nines) | 8.76 hours | 43.2 minutes |
| 99.95% | 4.38 hours | 21.6 minutes |
| 99.99% (four nines) | 52.56 minutes | 4.32 minutes |
| 99.999% (five nines) | 5.26 minutes | 25.9 seconds |
When every component is needed (serial), availabilities multiply. Three 99.9% components: 0.999 x 0.999 x 0.999 = 0.997003, which is 99.70%, or 0.002997 x 8,760 = 26.3 hours of downtime a year. Each added dependency spends some of the budget. Five 99.99% components: 0.99995 = 0.99950, about 99.95%, or 4.38 hours a year.
When any one replica is enough (parallel), the failure probabilities multiply instead. Two replicas of 99% each fail together with probability 0.01 x 0.01 = 0.0001, so the pair is 1 - 0.0001 = 99.99%, or 52.6 minutes a year. That result assumes the failures are independent. Two replicas on one host, one rack or one AZ fail together, and the arithmetic does not apply.
Worked example: sizing a tier for 1M requests per second
The service stage takes 40 ms on average. By Little's law (L = arrival rate x time in system), the number of requests in flight is 1,000,000 x 0.040 = 40,000. If one node can hold 1,000 requests in flight (an assumption for the arithmetic; measure yours), you need 40 nodes just to keep up, and N+1 means at least 41 so that one failure or one deploy does not drop you below capacity.
Spread the fleet across three availability zones and the rule tightens. Normally each zone carries one third of the traffic. Lose a zone and the two survivors carry one half each, so each must be sized for 50% of peak, which means total capacity of 3 x 50% = 150% of peak. Surviving an AZ loss costs half again as much hardware, which is why "N+1" for a zone is not "one extra server". See Load balancing for health checks and draining.
Deep dive
Where state lives
A stateless tier keeps nothing between requests that another replica would need, so any backend can serve any request and the load balancer is free to pick by load. State then lives in a few deliberate places: the database, a shared cache, a cookie or token the client carries. A sticky tier pins a client to one backend (an in-memory session, a WebSocket, a connection-oriented protocol), which makes balancing uneven and turns a backend failure into a lost session. Pinning by hash with minimal reshuffling when the pool changes is the job of Consistent hashing. Prefer stateless tiers with a shared store unless the protocol forces stickiness.
Fan-out and tail latency
Real requests are rarely one call. A page or API call fans out to many backends and waits for all of them. If each call is slow with probability p, the request is slow when any call is: 1 - (1 - p)n. With p = 0.01 (one call in 100 exceeds its p99):
| Fan-out n | P(at least one slow call) = 1 - 0.99n |
|---|---|
| 1 | 1% |
| 10 | 9.6% |
| 100 | 63.4% |
| 1,000 | 99.996% |
This is the example in Dean and Barroso, The Tail at Scale (Communications of the ACM, 2013): a service with a 1 ms median and a 1 s 99th percentile makes 63% of fan-out-of-100 requests take over a second. The paper's remedies are hedged requests (send a duplicate to a second replica after a short delay, take the first answer), tied requests, and cutting variability at the source. Hedging adds load and is only safe for idempotent calls; propagate a deadline on every call so a downstream stage does not keep working after the caller has given up. Those rules, with retry budgets and backoff, belong to Timeouts, retries and backoff. Latency percentiles and why to alert on the tail are covered in the Google SRE chapter Monitoring Distributed Systems.
Control plane and data plane
Every tier has two halves. The data plane carries user traffic and must be fast and boring. The control plane decides how: it pushes routes, backend lists, certificates, rate-limit policy and health results to the data plane. Design for the control plane being down: data-plane nodes should keep serving with their last known configuration (fail static) rather than dropping traffic because the config service is unreachable. The same split appears in CDN purges (CDN), in gateways (Proxies and gateways) and in traffic steering (Global traffic steering).
Failure domains
A failure domain is a set of things that fail together: a process, a host, a rack, an availability zone, a region, a cloud provider, or a shared dependency such as one DNS vendor or one certificate authority. Parallel replicas only multiply availability across different domains. AWS describes an AZ as one or more discrete data centres with independent power, networking and connectivity inside a region (AWS Regions and Availability Zones), which is why the 3-AZ sizing above is the common baseline and a second region is the next step. The cascading-failure chapter of the SRE book, Addressing Cascading Failures, shows how an overloaded domain pushes its load onto the survivors and takes them down too, which is why load shedding matters.
Caching layers and who owns freshness
Caches are stacked along the path, and each has its own control of how stale an answer may be. Most of the work in "why do I see old data" is finding which layer answered.
| Layer | Who decides freshness | Control | Page |
|---|---|---|---|
| Browser | the origin's response headers | Cache-Control: max-age, validators | HTTP caching |
| CDN edge and shield | origin headers, overridden by CDN rules | s-maxage, purge, surrogate keys | CDN |
| Reverse proxy or gateway cache | proxy config, usually honouring origin headers | proxy cache rules and TTLs | Proxies and gateways |
| Application cache (for example Redis) | your code | explicit TTLs and invalidation on write | your cache's documentation |
| Database buffer pool | the database engine | eviction policy, not a correctness control | your database's documentation |
"Cache everywhere" is also "invalidation everywhere": a write has to reach, or wait out, every layer that holds a copy.
Reading a failure from the status code
When a tier in the middle fails, the tier in front reports it. In HTTP semantics (RFC 9110 section 15.6), 502 means the gateway got an invalid response from upstream, 503 means the service is unavailable (often overload or draining), and 504 means the gateway timed out waiting. Which hop emitted it is in the response headers and logs; the table of causes is in Error messages.
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
| A client talks to one server | A name resolves to a steering layer, then an edge, then a pool of many servers | One server is both the capacity limit and a SPOF; each tier removes one of those |
| DNS round robin is load balancing | A health-aware load balancer behind a VIP, plus DNS or anycast for steering between sites | Round robin returns dead servers and cannot weigh load; resolvers and clients cache answers, so it also reacts slowly |
| Five nines for our service | Compute the product over every serial dependency, then ask which are redundant across failure domains | A service that is 99.999% on paper is capped by its least reliable required dependency |
| Scale up: buy a bigger box | Scale out behind a balancer, and shard the data when one primary is not enough | A bigger box is still one SPOF and has a ceiling; stateless tiers scale out cheaply, state needs partitioning |
| The average latency is the number to track | Track p50, p99 and p99.9 per hop | Fan-out turns a rare slow call into a common slow request |
| Add a load balancer and the SPOF is gone | The balancer is itself replicated (an HA pair or an anycast or ECMP fleet) | You moved the SPOF up one tier unless the balancer is redundant too |
In production
- Google documents a two-stage frontend: DNS steers users to a datacentre, then virtual IPs (VIPs) in that datacentre spread connections over servers (Load Balancing at the Frontend, Load Balancing in the Datacenter). Its network load balancer, Maglev, runs on commodity servers rather than dedicated hardware (Maglev, NSDI 2016).
- Netflix serves video from its own cache appliances placed inside ISP networks (Open Connect) and fronts its APIs with the Zuul gateway (Zuul 2: the Netflix journey to asynchronous, non-blocking systems).
- Cloudflare describes a layered edge: a per-datacentre load balancer that spreads connections over its servers (Unimog) and its own proxy that connects the edge to origins (Pingora).
- AWS customers commonly chain Route 53 for DNS, CloudFront as the CDN, an Application Load Balancer and a compute tier such as ECS. That is the table above with product names on each row.
- Availability targets are an explicit design input in the SRE approach: the chapter Embracing Risk frames a target as an error budget to be spent, not a number to maximise.
History and evolution
- 1985 Every host name lived in one
HOSTS.TXTfile distributed from a single site (RFC 952), which stopped scaling as the network grew; DNS replaced it (RFC 1034). - 1995 A popular site had one address and one machine; DNS round robin (described in RFC 1794) returned several addresses so load spread, but it could not detect a dead server and answers were cached.
- 1996 F5 Networks was founded and became a main vendor of hardware load balancers that watch health and hide many servers behind one virtual address.
- 1998 Akamai was founded to move content to servers near users, because far-away origins and congested links made pages slow.
- 2004 Software proxies took over the middle tier: HAProxy in the early 2000s and nginx (first public release 2004) did the work on commodity servers, with no appliance to buy.
- 2016 Envoy (open-sourced by Lyft) and the service-mesh pattern moved retries, timeouts and telemetry into a sidecar proxy beside every service, because microservice fan-out made per-service libraries unmanageable.
- Now Edge compute (for example Cloudflare Workers, 2017 onward) runs code at the PoP, and HTTP/3 (RFC 9114) is widely deployed; see W3Techs for the live adoption share rather than a frozen number.
- Next More logic at the edge, multi-cluster meshes spanning regions, and serverless back ends that scale to zero, all of which add hops to the budget rather than remove them.
Hands-on
Time each stage of a request with curl. The -w variables are cumulative from the start, so subtract neighbours to get each stage:
Subtract: DNS 2 ms (a local resolver cache), TCP handshake 57.862 - 1.964 = 55.9 ms, TLS 122.363 - 57.862 = 64.5 ms, waiting for the first byte 189.040 - 122.363 = 66.7 ms. The TCP number is one round trip to the edge (about 56 ms on this run), and TLS 1.3 costs about one more; this is the "60 ms" line of the budget. Your numbers will differ with location and time; this output is from macOS curl 8.7.1.
Ask the response which tiers touched it:
server names the front tier, age says the object has sat in a cache for 1,815 seconds (a hit), and the cf-ray suffix names the Cloudflare data centre that answered (ATL here). Other CDNs use x-cache or via instead; header names are vendor-specific.
Walk the DNS delegation to see the control-plane half of the path:
Three referrals (root, com, the zone's name servers) then the answer, which has a 300 s TTL: that TTL is how fast a steering change can take effect. Output is abbreviated; this is dig 9.10.6, and newer versions format the trailer lines slightly differently.
To find where on the path packets are lost or slow, use mtr (Linux, and brew install mtr on macOS) or traceroute; see ping and mtr for reading the output, and the Debugging by layer for the order to check things.
Gotchas
Interview questions
Show answer
The client opens a TCP connection to that address, then does a TLS handshake, then sends the request. The address is usually not a single server but a virtual IP (VIP) owned by a load balancer or a CDN edge, which then picks a real backend. So the first machine to see the request is a front tier whose job is to hide how many servers are behind it.
Follow-ups:
- What is a VIP? An address that belongs to a service rather than to one machine; the balancer answers for it and forwards to a healthy backend, see Load balancing.
- Could the address be a CDN? Yes, an anycast address advertised from many PoPs, see CDN.
- Where does the cost show up? TCP and TLS each cost a round trip to that first tier, so it should be near the user.
Show answer
Resolve the name (DNS, where a steering layer picks a region), connect to a nearby edge (CDN or anycast PoP, which terminates TLS and answers cacheable requests), reach a load balancer VIP, pass through a proxy or gateway (auth, routing, limits), call the service, which reads a cache and then the database. Each can fail: DNS or steering returns a dead region, the edge loses a PoP, the balancer or gateway is overloaded, the service times out, the cache is cold, the database fails over. The defence at each tier is redundancy across failure domains plus a timeout and a fallback.
Follow-ups:
- What is the common SPOF people forget? A single balancer, a single DNS vendor, or a shared dependency like one config service.
- How does the client learn a PoP died? Health checks and routing withdraw it; clients retry another address, see Global traffic steering.
- Which 5xx would the user see? 502 for a bad upstream response, 503 for unavailable, 504 for upstream timeout, see Error messages.
Show answer
Browser for static assets with long max-age and versioned URLs. CDN for public, cacheable responses and for hiding the origin. A reverse-proxy or gateway cache for hot shared responses, an application cache (Redis or similar) for computed results and database reads, and the database's own buffer pool for pages. Put each class of data at the layer closest to the user where it is still correct to share it, and never cache per-user content at a shared layer.
Follow-ups:
- How do you invalidate? Versioned URLs for assets, TTLs for tolerable staleness, purge by tag for must-change-now, write-through or delete-on-write for the app cache; see CDN.
- What is the main risk? Stale or wrongly shared data; each added layer adds an invalidation path.
- What decides how long to keep it? How stale users can tolerate, not how big the cache is.
Show answer
For services that are all required, multiply their availabilities: three at 99.9% give 0.9993 = 99.70%, about 26 hours a year of downtime. For redundant replicas that fail independently, multiply the failure probabilities: two 99% replicas give 1 - 0.012 = 99.99%. So reduce the number of serial dependencies, make the weakest ones redundant across failure domains, and degrade gracefully when a non-critical dependency is down so it stops being "required".
Follow-ups:
- How many minutes is four nines? 52.56 minutes a year, 4.32 minutes in a 30-day month.
- What breaks the parallel formula? Correlated failures: the same rack, AZ, deploy or shared dependency.
- Can a service be more reliable than its dependencies? Yes, with caching, fallbacks and redundancy, so a dependency's outage is not a user-visible outage.
Show answer
At the CDN edge or the load balancer, because the handshake costs round trips and a nearby terminator makes them short, and because certificates and cipher policy are managed in one place. Whether to re-encrypt to the backend depends on the threat model: inside a trusted network some teams run plain HTTP behind the balancer; regulated or zero-trust designs encrypt every hop, often with mutual TLS. Encrypting costs CPU but removes the plaintext hop.
Follow-ups:
- What do you lose by terminating at the edge? The edge sees plaintext, so you trust that provider with it.
- What is passthrough? The balancer forwards the encrypted bytes without decrypting, which keeps end-to-end encryption but removes layer 7 routing there.
- Where is the handshake explained? TLS.
Show answer
Start with the numbers. At 40 ms per request, Little's law gives 40,000 requests in flight; divide by what one node sustains to get N, add N+1, and across 3 AZs provision 150% of peak so one AZ can fail. Steer with DNS or anycast to the nearest healthy region, absorb cacheable traffic at a CDN, spread the rest over a redundant balancer tier, keep services stateless, and shed load before queues grow. Budget availability as a product: with four nines you have about 4.3 minutes a month, so every serial dependency must be redundant or degradable, and the control plane must fail static.
Follow-ups:
- What are the failure domains? Host, rack, AZ, region, provider, and shared dependencies such as DNS and certificates.
- What is headroom for? Absorbing a lost AZ, a deploy and a traffic spike at once.
- How do you protect against overload? Rate limiting and load shedding, see Rate limiting and load shedding.
Show answer
A request that calls n backends is slow if any call is slow, so the chance is 1 - 0.99n when each call is slow 1% of the time: 9.6% for 10 calls and 63.4% for 100. The tail of the parts becomes the typical case of the whole. Mitigate with hedged requests (a duplicate to another replica after a delay, first answer wins), per-call timeouts and deadlines that propagate, fewer or smaller fan-outs, and reducing variance (queueing, GC, noisy neighbours). Hedging is safe only for idempotent calls and must be rate-limited so it does not add load during an incident.
Follow-ups:
- What is a hedged request? A second copy sent after the typical latency has elapsed; see The Tail at Scale.
- Why propagate a deadline? So downstream work stops once the caller has given up, see Timeouts, retries and backoff.
- How do retries interact? Retries multiply load under failure; use budgets and backoff.
Show answer
Reproduce from a client and split the time with curl -w (DNS, TCP, TLS, first byte), which says whether the cost is name lookup, distance, handshake or server think time. If it is server time, go to distributed tracing to see the span per service, and per-service metrics: rate, errors and duration, or the four golden signals (latency, traffic, errors, saturation) from the SRE book chapter on monitoring. Compare percentiles, not means, and check whether the slowness is one region, one backend or one dependency.
Follow-ups:
- How do you tell network from application? Large TCP connect time points at the path or distance; large time-to-first-byte with a fast connect points at the server.
- What if curl is fast but users are slow? Test from their location and network; check DNS steering and cache hit status in the response headers.
- Where is the checklist? Debugging by layer.
Fun facts
- Light in fibre travels at about two thirds of its speed in vacuum, so Virginia to Sydney can never beat a 157 ms round trip at the floor, however fast the routers are.
- Four nines allows about 4.3 minutes of downtime in a 30-day month, less than many deploys take.
- With one slow call in 100, a request that fans out to 100 calls is slow 63.4% of the time, the headline number of The Tail at Scale.
- DNS round robin was written up as an RFC in 1995 (RFC 1794), long before health-aware balancers.
- Losing one of three zones means each survivor must run at 50% of the original peak, which is why zone-proof capacity is 150% of peak.
Cheat sheet
| Number | What it is |
|---|---|
| 0.5 ms | same data centre round trip (rough) |
| 1 to 2 ms | cross-AZ round trip in one region (rough) |
| 55 ms | Virginia to Ireland round-trip floor (about 5,500 km at 200,000 km/s in fibre) |
| 157 ms | Virginia to Sydney round-trip floor (about 15,700 km) |
| 99.9% x 3 | 99.70% serial, 26.3 hours a year down |
| 1 - 0.012 | two independent 99% replicas, 99.99% |
| 1 - 0.99100 | 63.4% of 100-way fan-outs hit a p99 call |
| 1,000,000 x 0.040 | 40,000 requests in flight (Little's law) |
| 150% of peak | capacity needed to lose 1 of 3 zones |