Debugging by Layer - From Symptom to Command
- Debug by bisecting the stack: name (DNS), path, port, TLS, app. Each rung is one command with one pass or fail meaning, and the first rung that fails is where you look.
- Refused, timed out and reset are three different facts. Refused: a machine answered with a RST at once, so the path works and nothing is listening. Timed out: silence, so something dropped the packets or the host is gone. Reset: an established connection was torn down.
- Each tool answers one layer.
pingworking does not mean the app works, andpingfailing does not mean the host is down. - Measure before you guess.
curl -wsplits one request into DNS, connect, TLS, wait and download, so "slow" becomes a number on one phase. - Layers leak: an MTU black hole looks like an application hang, and
digcan work while your program cannot resolve the name.
Mental model
A plumber tracing a leak does not start by tearing out walls. They open a joint halfway along the pipe, see whether water arrives, and now know which half has the leak. They halve again. A few joints later they are holding the leaky section. Each test is cheap and each answer removes half the pipe from the problem.
Where it breaks: a pipe has one leak at one place, and water only flows one way. A network can have two faults at once, and a fault in the return direction is invisible to a test that only watches the forward direction.
This is git bisect over the stack. The bottom (cable, address, route) is the known-good commit and the user's symptom is the known-bad one. Each rung of the ladder is a test you run on a midpoint, and the first failing rung tells you which range to search.
Where it breaks: git bisect assumes a bug that, once present, stays present. Layers are not independent in that way. An MTU black hole passes the name lookup, the ping and the port check, because they all use small packets, and fails only when a big packet shows up. It surfaces at the TLS or HTTP rung even though the cause is at the network layer. When the ladder gives an answer that contradicts the one below it, suspect size or timing, not the rung.
How it works
The ladder follows the order a client actually does things (the layer model in practice): resolve a name to an address, send packets to that address, open a connection to a port, negotiate TLS, send the request. Every step needs the step before it, so a failure at rung N means rungs 1 to N-1 already passed and you do not have to re-check them.
Two honest caveats about the ladder. Rung 2 is the only rung that depends on ICMP, and many firewalls drop ICMP (ICMP and ping has the mechanism), so a failed ping with a working rung 3 means ICMP is filtered, not that the path is broken. If you suspect that, go straight to rung 3: connecting to the real port tests the path and the port at once. And rung 1 has two answers, because dig talks to a DNS server directly while programs ask the operating system (see why dig works but the app fails).
Try the ladder on a symptom. Pick what was reported, run the command it gives you, and choose the output you saw. Each answer either names the next command or ends in a likely cause and the page that explains it. A table cannot branch like this, and the order of the rungs is the point.
dig +short example.comdig asks a DNS server directly. It skips /etc/hosts and nsswitch, so it tests the name server, not what your program sees.
Worked example: "the site is slow"
Do not guess which layer. Ask curl for the timestamps of each phase of one request:
This is one real run from a Mac on home Wi-Fi to example.com; your numbers will differ. curl reports each value as seconds since the start of the request (curl manual, --write-out), so the time spent in each phase is the difference between neighbours:
| Phase | curl variable | Cumulative (s) | This phase (ms) | What it measures |
|---|---|---|---|---|
| DNS | time_namelookup | 0.003873 | 3.9 | name to address, here a cache hit |
| TCP connect | time_connect | 0.056396 | 52.5 | SYN out, SYN-ACK back: one round trip |
| TLS | time_appconnect | 0.128545 | 72.1 | handshake, about one more round trip plus crypto |
| Wait for first byte | time_starttransfer | 0.198048 | 69.5 | request sent, server thinks, first byte back |
| Download | time_total | 0.198444 | 0.4 | rest of the body |
The four phases add up to the 198.4 ms total. A ping to the same host in the same session averaged 58.2 ms over three packets, so the 52.5 ms connect phase is about one network round trip, as it should be. Here nothing is wrong. The point is the reading: if connect were 2 s, the path or the SYN is the story; if tls were 2 s, look at the handshake; if the wait were 2 s, the network delivered your request promptly and the server is the slow part. That one command usually replaces an hour of arguing between the network team and the app team.
Symptom to first command
This table inverts the ladder: start from what was reported and jump to the command most likely to turn it into a fact. The last column names the page that explains the mechanism, so this page does not have to. Commands are given for Linux; the macOS equivalents are in Hands-on.
| Symptom | First command | What the output means | Likely cause | Owner page |
|---|---|---|---|---|
| Works locally, fails in prod | curl --resolve example.com:443:PROD_IP https://example.com/ and dig @RESOLVER example.com from both places | Works with --resolve: the server and app are fine and the name points elsewhere | DNS record, security group or certificate differs between environments | dig and DNS debugging |
| The site is slow | curl -w phases (see the worked example) | The largest phase is the layer to chase | DNS, path RTT, TLS handshake or server think time | ss, nc, lsof and ip, TCP performance |
| Slow only on the first request | curl -w with the URL given twice in one command, so the second request reuses the connection | First request pays dns, connect and tls; the second shows them near zero | Cold DNS cache, new TCP and TLS handshake, cold server cache | ss, nc, lsof and ip |
| Works by IP, not by hostname | dig +short HOST, then getent hosts HOST | dig answers but getent does not: the OS resolver path differs from dig | /etc/hosts, nsswitch.conf order, search domains, wrong resolver in /etc/resolv.conf | DNS |
| Connection refused | ss -ltnp on the server | No LISTEN line for that port and address means no process is waiting there | Service down, bound to 127.0.0.1 or ::1 only, wrong port, or a firewall REJECT rule | error messages to cause |
| Connection timed out | nc -zv HOST PORT, then tcpdump -nn port PORT on the server | SYN never shows on the server: dropped on the way. SYN shows but no reply: the server side drops it | Firewall or security group DROP, wrong route, dead host | firewalls, security groups vs NACLs |
| Connection reset mid-transfer | tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0' | Which side sent the RST, and after how much idle time | Peer crashed, load balancer idle timeout, half-open connection after a reboot | TCP state machine |
| SSH or a transfer hangs on large output, small works | ping -M do -s 1472 HOST | Small pings pass, this one fails or says the message is too long: path MTU is under 1500 | PMTUD black hole: ICMP fragmentation-needed is filtered; clamp the MSS | MTU, MSS and fragmentation |
| Intermittent timeouts or loss | mtr -rwc 100 HOST, from both ends | Loss that persists to the last hop is real; loss at one hop that disappears after it is ICMP rate limiting | Congested or flapping link, bad NIC or cable | ping and mtr |
| Slow to one region only | mtr -rwc 50 to that region, then to a healthy one | Where the latency jumps between the two traces | Congested or indirect transit or peering path | ping and mtr |
| Cannot assign requested address under load | ss -Htan state time-wait | wc -l and cat /proc/sys/net/ipv4/ip_local_port_range | Thousands of TIME_WAIT entries to one destination, near the size of the port range | Ephemeral port exhaustion from short-lived connections to a single address and port | port exhaustion and TIME_WAIT |
| Outbound calls fail or stall from a private subnet under load | conntrack -C against sysctl net.netfilter.nf_conntrack_max; on AWS the NAT gateway ErrorPortAllocation metric | Count at the limit, or the kernel log says nf_conntrack: table full, dropping packet | Connection-tracking table full, or NAT gateway out of source ports | NAT gateways and egress |
| TLS works in curl, fails in the app | openssl s_client -connect HOST:443 -servername HOST, then check which CA bundle the app uses | Verify return code other than 0 names the failure | App's CA bundle lacks the issuer, SNI not sent, or clock skew makes the certificate look expired | TLS |
| UDP-only loss: DNS or QUIC fails, TCP is fine | dig +tcp HOST versus plain dig HOST, and tcpdump -nn udp port 53 | TCP answer arrives but UDP does not: UDP is blocked or large UDP answers are lost | Firewall allows TCP only, or fragments are dropped | error messages to cause |
| SYN arrives, no SYN-ACK seen back | tcpdump -nn host PEER on both ends, and mtr from each end | Packets seen in one direction only | Asymmetric routing through a stateful firewall that never saw the first half | firewalls |
| Intermittent 5xx under load | curl -s -o /dev/null -w '%{http_code}\n' in a loop, then the proxy's log for the upstream error | 502, 503 and 504 point at different upstream failures | Upstream refusing, resetting or timing out; no healthy backend; deliberate shedding | error messages to cause, proxies and gateways |
| Exactly 5 s delay on name lookups in Kubernetes | time getent hosts NAME and cat /etc/resolv.conf | options ndots:5 plus a long search list makes one lookup into several queries; a lost UDP query is retried after the 5 s default timeout | Search-domain expansion, or a dropped DNS query | DNS |
| Reachable by public IP, not private IP, inside a VPC | curl -v http://PRIVATE_IP:PORT/, then check the route table, security group and NACL | Timeout means a drop on the way; refused means you reached the host and nothing listens | Security group source rule, NACL, or a route that sends private traffic elsewhere | VPCs, subnets and routing |
Deep dive
The four outcomes of a connect
When a client calls connect(), the kernel sends a SYN and exactly one of four things happens. Telling them apart is the most useful single skill in this module, because each points at a different cause.
| Outcome | On the wire | errno | Typical cause |
|---|---|---|---|
| Open | SYN, then SYN-ACK | none, connect returns 0 | A process is listening and nothing blocks the path |
| Refused | SYN, then RST at once | ECONNREFUSED | Nothing listening on that address and port, or a firewall REJECT rule |
| Timeout | SYN repeated, no reply | ETIMEDOUT | Firewall DROP, a route that goes nowhere, host down or unplugged |
| Unreachable | SYN, then ICMP destination unreachable | EHOSTUNREACH or ENETUNREACH | No route to the network or host, or an administrative prohibit |
| Reset (established) | RST during an open connection | ECONNRESET on read, EPIPE on a later write | Peer crashed, idle timeout in a load balancer or NAT, half-open connection |
The kernel does not give up quickly on silence. On Linux, tcp_syn_retries defaults to 6, "which corresponds to roughly 127 seconds" on the classic schedule (tcp(7)); recent kernels, where tcp_syn_linear_timeouts defaults to 4, give up after about 131 seconds (ip-sysctl). On the macOS machine used for this page, sysctl net.inet.tcp.keepinit reports 75000, a 75 second connect timeout. Programs that need a faster verdict set their own connect timeout; the retry arithmetic is in timeouts, retries and backoff. The full errno vocabulary, including EADDRINUSE and DNS errors, is in error messages to cause; this page only needs the four connect outcomes.
Two details that trip people up. First, a firewall rule that rejects instead of drops produces a fast failure, not a timeout: iptables REJECT replies with an ICMP port unreachable by default or, with --reject-with tcp-reset, a RST (iptables-extensions(8)). Either way the client gets a quick ECONNREFUSED, so a fast refusal does not prove nobody is listening. Second, a timeout is evidence that packets are lost in one direction or the other, and the SYN-ACK can be lost on the way back just as easily as the SYN on the way in.
Reset on an established connection
A RST in the middle of a conversation means the peer, or a device acting for it, decided the connection no longer exists. Reproduced on this Mac with a server that closes with a zero linger time, which makes the kernel send a RST instead of a FIN:
In production the usual culprits are a crashed or restarted process, a load balancer or NAT dropping a connection it considered idle, and a half-open connection (the peer rebooted and has no memory of you). The load balancer case is worth checking against the documented timeouts: an Application Load Balancer's idle timeout defaults to 60 seconds (ELB docs) and a Network Load Balancer's TCP idle timeout is 350 seconds (NLB docs). A reset that arrives right after exactly that much idle time is a strong hint. The states involved are on the TCP state machine page.
Why dig works but the app fails
dig builds a DNS query and sends it to a DNS server. It does not read /etc/hosts, and it does not follow the lookup order the system is configured with. Programs call getaddrinfo, which on Linux consults the hosts: line of /etc/nsswitch.conf (nsswitch.conf(5)) and on macOS goes through the system resolver. So dig can return an address while the program resolves a different one, or nothing. Ask the operating system the way a program does, and compare:
Here both agree, and the OS path also returns IPv6 addresses. When they disagree, read /etc/hosts, the hosts: order, and the search and ndots lines in /etc/resolv.conf before blaming DNS. Resolver mechanics belong to DNS.
Small works, large hangs: the MTU black hole
TCP handshakes, DNS answers and short requests fit in small packets, so rungs 1 to 3 pass. A bulk response, or a TLS server certificate flight, fills packets to the full size, and if some link on the path has a smaller MTU and the ICMP "fragmentation needed" message that should tell the sender never arrives, those packets vanish and the connection stalls. Confirm it by sending a ping that may not be fragmented. The payload is 1500 minus 20 bytes of IP header minus 8 bytes of ICMP header, which is 1472:
On this path the 1500-byte packet gets through, so there is no black hole here. On a path with a smaller MTU, a working small ping and a failing 1472-byte one is the diagnosis; the Linux form is described in ping(8) and I did not capture its failure output. The fix, clamping the MSS, and the full PMTUD story live on MTU, MSS and fragmentation.
Intermittent faults
"Intermittent" is not a diagnosis, it is a sampling problem. A single traceroute is one sample. Run mtr -rwc 100 HOST so each hop gets 100 probes, and run it from both ends, because the return path can differ from the forward path. Loss that starts at a hop and continues to the final hop is real. Loss at one hop with clean hops after it is that router deprioritising its own ICMP replies, not dropping your traffic. When the tools disagree with the user, capture packets at one end and look for retransmissions; tcpdump and Wireshark shows how, and TCP performance explains what the retransmissions mean.
Debugging with no tools installed
On a minimal container image there may be no dig, nc or curl. Linux exposes the kernel's socket table as text in /proc/net/tcp, and bash can open a TCP connection itself with a /dev/tcp/HOST/PORT redirection (bash manual, Redirections). Illustrative, Linux bash, not captured on this Mac:
The exit status gives the same three-way answer: success, a quick failure (refused), or timeout killing it (silence).
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
ping fails, so the host is down | Many hosts and cloud security groups drop ICMP while serving traffic normally | ICMP is a separate protocol with its own rules; AWS security groups, for example, allow no inbound ICMP until you add a rule |
traceroute shows the path your packets take | It shows one sampled forward path; load balancing across equal-cost routes and a different return path are invisible | Each probe may hash to a different link, and the reverse path is chosen by other routers |
| Refused means the server is down | The process may be up on a different port, or listening on IPv6 only | A listener is bound to one address family and port; see the capture in Gotchas |
| Each layer fails independently, test them in isolation | A path MTU problem surfaces as a TLS or HTTP hang | Layers pass sizes and timing to each other; a bug in one can only be seen through the next |
In production
- Google SRE treats troubleshooting as a method, not a bag of tricks: form a hypothesis, test it, and narrow the search, rather than changing things at random (SRE Book, Effective Troubleshooting). The ladder on this page is that method specialised to networking.
- AWS offers the cloud version of testing each hop. VPC Flow Logs record, per flow, whether the traffic was
ACCEPTed orREJECTed (VPC Flow Logs), which tells you whether a security group or NACL dropped it. Reachability Analyzer reasons over the configured route tables, security groups and gateways to explain why a path is or is not reachable (Reachability Analyzer). - Kubernetes pods get a resolver configuration with a search list and
ndots, and the documentation showsndots:5in the generatedresolv.conf(Kubernetes DNS for Services and Pods). That is the source of the lookup fan-out in the symptom table. - Runbooks in most operations teams are this ladder written down per service: which command, what a pass looks like, and who to page when a rung fails.
History and evolution
- 1983 Mike Muuss writes
ping, an echo test for whether a host answers and how long the round trip takes, and names it after the sound of sonar (Muuss's own account). - 1987
traceroutearrives (Van Jacobson; the date is the commonly cited one), using the IP time-to-live field so each router along the path identifies itself. It answered a questionpingcould not: where along the path things break (mechanism). Tools stayed one per layer (network, sockets, names, packets) because the stack they inspect is layered. - Now eBPF tools such as bpftrace and Cilium's Hubble observe sockets, packets and system calls in one place, so one probe can cross layers that the classic tools see separately.
- Next Cloud networks are configuration, not cables, so more of the ladder is answered by analysis of that configuration (Reachability Analyzer is an example) before any packet is sent.
Hands-on
Run the ladder yourself against any host you are allowed to probe. Outputs below are real captures from macOS, abridged, against example.com.
Rung 4 and 5, with the handshake summarised:
(That output was filtered with grep to four lines.) The ttl=48 in the ping is the TTL left on the reply, not the number of hops out; many systems start replies at 64, so roughly 16 hops on the way back, a hint only. Now produce the failures on purpose so you recognise them:
Notice the refusal returned instantly and the timeouts waited out their limits, which is the distinction in the four-outcomes figure. Then reproduce the IPv6-only trap: start a server bound to the IPv6 loopback and connect on both address families.
The server is up and refused is the right answer for IPv4: the listener is on IPv6 only. Stop the server with kill %1.
macOS and Linux side by side
| Task | macOS | Linux |
|---|---|---|
| Resolve like a program | dscacheutil -q host -a name HOST | getent hosts HOST |
| Test a port | nc -zv -G 5 HOST PORT (-G is the connect timeout) | nc -zv -w 5 HOST PORT (-w is the timeout) |
| Who is listening | lsof -nP -iTCP -sTCP:LISTEN | ss -ltnp |
| Which route and gateway | route -n get IP | ip route get IP |
| Count TIME_WAIT | netstat -an -p tcp | grep -c TIME_WAIT | ss -Htan state time-wait | wc -l |
| Ping without fragmenting | ping -D -s 1472 HOST | ping -M do -s 1472 HOST |
| Install mtr | brew install mtr | your package manager, package mtr |
ss and ip do not exist on macOS, and mtr is not installed by default on either system. dig, nc, curl and openssl exist on both, though macOS ships older versions (this Mac has DiG 9.10.6 and curl 8.7.1 with LibreSSL), so flags and output can differ slightly. Per-tool detail is in ping and mtr, dig and DNS debugging, ss, nc, lsof and ip and tcpdump and Wireshark.
Gotchas
Interview questions
Show answer
First measure, do not guess. Reproduce it with curl -w for the timing of each phase, or open the browser's network waterfall, so I can see DNS, connect, TLS, wait for first byte and download as separate numbers. The biggest phase tells me which layer to investigate: DNS, path, handshake or the server itself.
Follow-ups:
- What if only the first request is slow? DNS cache miss, a new TCP and TLS handshake, or a cold server cache; a second request on a reused connection skips the first two.
- What if only one region is slow? Compare
mtrto that region against a healthy one and look for the hop where latency jumps, which points to a transit or peering problem.
Show answer
Refused: the host answered my SYN with a RST at once, so the machine is reachable and nothing is listening on that port, or a rule rejected it. Timed out: nothing came back, so the packets were dropped or the host is gone. Reset: a connection that was working was torn down by a RST.
Follow-ups:
- Which one is a firewall DROP? Timed out, because a drop is silent.
- Which one is a crashed process? Refused if it crashed before the attempt; reset if it crashed mid-connection.
Show answer
ICMP is filtered somewhere on the path or on the host, while TCP to the web port is allowed. AWS security groups, for example, drop inbound ICMP unless a rule allows it. So I trust the TCP result for "can I reach the service" and treat the failed ping as a firewall policy, not an outage.
Follow-ups:
- What about the reverse, ping works but curl fails? The host is reachable but the port, TLS or the app is the problem; move up the ladder to
nc,opensslandcurl -v.
Show answer
Use curl -w and read the first two numbers. A big time_namelookup is DNS, a big time_connect minus it is the network round trip to the server, time_appconnect minus connect is TLS, and a big gap before first byte is the server. One command, four layers.
Follow-ups:
digis fast but the app is slow, why?digskips/etc/hosts, search domains and the resolver the app uses; time the lookup the way the app does it, withgetent hostson Linux.
Show answer
Collect a large sample: mtr -rwc 100 to the target, from both ends, and read it carefully. Loss that persists to the last hop is real; loss at one hop with clean hops after it is that router rate limiting its ICMP replies. For proof, capture with tcpdump and look for retransmissions.
Follow-ups:
- Why both directions? The return path can be different, and the loss may only be on it.
- Why can a middle hop show loss that is not real? Routers often deprioritise replies they generate themselves compared with the traffic they forward.
Show answer
Classic path MTU black hole. Small packets pass, but full-size ones are dropped by a link with a smaller MTU, and the ICMP "fragmentation needed" message that should shrink the sender's packets is filtered. I confirm with a don't-fragment ping at 1472 bytes payload (1500 minus 28), and fix it by clamping the MSS at the edge or allowing the ICMP message through.
Follow-ups:
- Why does the handshake still work? SYN and SYN-ACK are tiny; only the bulky data and the certificate flight hit the limit.
- Where does 28 come from? A 20-byte IPv4 header and an 8-byte ICMP header.
Show answer
Work up the ladder from inside the pod. DNS first: the pod's resolv.conf has a search list and ndots:5, so a name can turn into several queries and a lost one costs a 5 second retry. Then the path and port: security groups, NetworkPolicy and source NAT on the node can all differ from my laptop's. I use getent hosts, nc -zv and curl -v from a debug container in the same namespace.
Follow-ups:
- Why does source NAT matter? The target sees the node's address, not the pod's, so an allow-list keyed on the pod address fails.
- What tool do you reach for in a pod with no tools? An ephemeral debug container, or bash's
/dev/tcp.
Show answer
Use what the kernel and the shell already provide. On Linux I read /proc/net/tcp to see which sockets exist and their states, and I test a port with bash's /dev/tcp/host/port redirection inside timeout: success means open, a fast failure means refused, and the timeout firing means silence. That is the same three-way answer as nc.
Follow-ups:
- Does this work on macOS? Not the
/procpart; usenetstatandlsofthere, and I did not verify/dev/tcpon macOS's bash.
Fun facts
pingwas not an acronym. Muuss named it for the sonar sound of an echo; the backronym "Packet InterNet Groper" was attached later (Muuss's account).- A
curlcommand with the same URL twice reuses one connection: on this Mac the second request showedtime_connect 0.000000andnum_connects 0, against0.053445and1for the first. - Port 9 is the "discard" service (RFC 863), which almost nobody runs any more, so connecting to it on your own machine is a harmless way to see a refusal.
Cheat sheet
- 127 s - Linux connect timeout on silence, classic schedule:
tcp_syn_retriesdefault 6 (tcp(7)); about 131 s on recent kernels (ip-sysctl) - 75 s - macOS connect timeout,
net.inet.tcp.keepinit= 75000 ms (measured on the machine used here) - 1472 - largest DF ping payload for a 1500-byte path: 1500 - 20 - 8
- 5 s - default resolver query timeout, retried on loss (resolv.conf(5))
- ndots:5 - the Kubernetes default in pod
resolv.conf
The ladder in one line: name, path, port, TLS, app. First failure is the layer. Refused is fast and means "reachable, not listening"; timeout is slow and means "dropped"; reset means "was open, got cut".