Failure Lab - Reproduce Each Network Failure Locally
- You learn an error by causing it. Every failure here is reproduced on loopback, in your own container or against a documentation host, so nothing touches a real service.
- The core lab is refused versus timeout. No listener answers a SYN with an RST at once. A firewall
DROPanswers with silence and the client retransmits. A firewallREJECTanswers with a reset or an ICMP error. - Each lab has the same shape: goal, command, symptom, what to look at, and the undo command next to every command that changes the system.
- macOS and Linux differ. The macOS labs below were run for real (dated). Anything needing root or Linux (
iptables,tc netem,ip link,nstat) is marked illustrative.
Mental model
A fire drill. Nobody learns the evacuation route by reading the sign; they learn it by walking it once when nothing is burning. Each lab here is one drill: you trigger the exact alarm (a closed door, a silent phone, a hang-up) on a quiet day, so that on a bad day the symptom is familiar.
Where it breaks: a drill is staged and complete. A real failure arrives with noise from everything else on the network, and several of these causes look alike from the client side, so the lab teaches the signature, not the diagnosis.
A fault-injection test suite. Each lab is a unit test with a known input (a closed port, a dropped packet, an expired certificate) and an expected exception (ECONNREFUSED, ETIMEDOUT, a verify error). Run them once and you have a reference table to compare production errors against.
Where it breaks: unit tests assert a single outcome. These labs also have a time dimension (the timeout is the symptom), so you need a stopwatch, curl -w or a packet capture, not only an exit code.
How it works
The setup, and what is safe
Everything runs against 127.0.0.1, ports you pick (8000 to 9999 here), documentation hostnames, or a Linux container or VM you own. Three rules keep it safe.
- Never run
iptables,tcorpfctlrules on a production host. They apply to all matching traffic on the machine, including your SSH session. - Every system-changing command has its undo next to it. Run the undo before you close the terminal.
- Use only hosts meant for failure.
badssl.compublishes deliberately broken certificates,.invalidis a reserved top-level domain that never resolves (RFC 6761 section 6.4), and192.0.2.1is in a documentation range that is not routed (RFC 5737).
The tools are the ones in this module: ss, nc, lsof and ip for sockets, tcpdump and Wireshark for the wire, curl and openssl for the client, dig for DNS, and ping and mtr for reachability. What each error means is on Error Messages to Cause; the layer-by-layer method is on Debugging by Layer.
The lab index
| # | Failure | Where | Symptom | Undo |
|---|---|---|---|---|
| 1 | Connection refused | macOS and Linux, ran on macOS | curl: (7) in 0 ms | nothing to undo |
| 2 | Connect timeout (DROP vs REJECT) | Linux, illustrative; macOS pfctl | curl: (28) after the timeout | iptables -D, pfctl -d |
| 3 | Read timeout | macOS and Linux, ran on macOS | curl: (28) with connect time near 0 | stop the server |
| 4 | Connection reset | macOS and Linux, ran on macOS | curl: (56), ECONNRESET | nothing to undo |
| 5 | Address in use and TIME_WAIT | macOS and Linux, ran on macOS | EADDRINUSE | stop the server |
| 6 | DNS NXDOMAIN, SERVFAIL, stall | macOS and Linux, ran on macOS | status: NXDOMAIN, SERVFAIL, timeout | nothing to undo |
| 7 | TLS cert mismatch and expired | macOS and Linux, ran on macOS | curl: (60) | nothing to undo |
| 8 | Loss and latency | Linux tc netem, macOS dnctl; illustrative | RTT and loss in ping | tc qdisc del, dnctl flush |
| 9 | PMTUD black hole | Linux, illustrative; macOS local variant ran | small works, large hangs or errors | ip link set mtu back |
| 10 | Port exhaustion | Linux, illustrative | EADDRNOTAVAIL | restore the sysctl |
| 11 | Slow receiver and zero window | macOS and Linux, ran on macOS | send() blocks, win 0 | stop both ends |
Hands-on
Dated output below was captured on macOS (Darwin 27.0.0, curl, nc, dig 9.10.6, Python 3.14) on 2026-10-14. Output is abridged. Blocks marked "illustrative" were not run on this machine because they need root or Linux; their output is typical, not captured.
Lab 1: connection refused
Goal: see what an RST in reply to a SYN looks like to the client. Nothing listens on port 9999.
after 0 ms is the signature: the kernel replied immediately. With curl -v the line reads connect to 127.0.0.1 port 9999 ... failed: Connection refused. To watch the wire (needs root, illustrative), capture loopback and run the curl again:
A SYN, then [R.] (RST plus ACK) with no handshake in between. That is RFC 9293 section 3.5.2 reset generation. Now start a listener to see the success case, and find it:
Undo: kill %1 (or the PID that lsof printed). Pick a port nobody else uses: on this machine port 8000 was already taken by another listener, which is itself the next lab.
Lab 2: connect timeout, with a firewall that drops or rejects
Goal: produce a timeout and a refusal against the same port, so the difference is something you have felt. This needs a Linux box, VM or container you own, and root. Illustrative; do not run it on a host you cannot afford to lock yourself out of.
A listener exists and the port is open, yet the connect hangs until curl's own --connect-timeout. Without that flag, Linux gives up on its own after the SYN retries, governed by net.ipv4.tcp_syn_retries (default 6, 127 s on the classic schedule and about 131 s on recent kernels per the kernel ip-sysctl reference); see Retransmission and timers. In tcpdump -i lo -nn 'tcp port 8000' you see the same SYN repeated with the gap doubling (illustrative):
Now swap the rule so the same port is refused instantly, two ways. Remove the DROP first:
Undo is the -D line that mirrors each -A; iptables -S INPUT must show none of your rules. The --reject-with options are documented in iptables-extensions(8).
macOS substitute. macOS has no iptables; the equivalent is pf. This replaces the active pf ruleset, so only do it on a personal Mac and undo it immediately (illustrative, needs root, not run here):
block drop is the DROP analogue; block return is the REJECT analogue (RST for TCP, ICMP unreachable for UDP). See pfctl(8) and pf.conf(5).
Lab 3: read timeout
Goal: a connection that opens fine and then goes silent. This is a different failure from lab 2, and curl's timing fields show it. The server below accepts and then sleeps.
Same exit code 28 as lab 2, but the message differs (Operation timed out ... with 0 bytes received, not Failed to connect ... Timeout was reached) and time_connect is near zero: the handshake finished, the peer never answered. The distinction is explained in Connect timeout versus read timeout. Undo: kill %1.
Lab 4: connection reset
Goal: get ECONNRESET. The server below reads one request, then closes with SO_LINGER set to zero, which makes close() send an RST instead of a FIN.
A Python client sees the same thing as an exception: ConnectionResetError: [Errno 54] Connection reset by peer (errno 54 on macOS, 104 on Linux). The other classic route needs no code: run nc -l 9000 in one terminal, connect with nc 127.0.0.1 9000 in another, type a line, then kill the listener with the client still attached; the next write from the client gets a reset. (BSD nc as on macOS takes -l 9000; some Linux nc builds need -l -p 9000, see nc(1).) In a capture the end of the exchange is a bare [R] on an established connection, unlike lab 1 where the RST answers a SYN. Reset semantics are on Reset: the peer said abort.
Lab 5: address in use and TIME_WAIT
Goal: see EADDRINUSE two ways. First, two servers on one port:
(errno 48 on macOS, 98 on Linux.) Second, the case where nothing is running but the restart still fails: a server that closes first leaves a socket in TIME_WAIT on its own port, and a plain bind() is refused until it expires. This script closes a connection from the server side and then rebinds, with and without SO_REUSEADDR:
The second run passed even though a TIME_WAIT entry existed, because SO_REUSEADDR lets a listener bind a port that has only TIME_WAIT leftovers. Linux: ss -tan state time-wait '( sport = :9400 )'. Why the state exists, and why the fix is not to shorten it, is on Port exhaustion and TIME_WAIT.
Lab 6: DNS failures
Goal: three different DNS outcomes. .invalid never resolves, so it gives a clean NXDOMAIN:
SERVFAIL with a safe, real example: dnssec-failed.org is a zone with a deliberately broken DNSSEC signature, so a validating resolver refuses it. Asking again with +cd (checking disabled) shows the data is there and only validation failed:
(This depends on your resolver validating DNSSEC; one that does not returns NOERROR to both.) Last, the stall: send a query to an address that is in a documentation range and will never answer.
That took the full 2 s. A system resolver with a dead nameserver in /etc/resolv.conf stalls for its own timeout (5 s per attempt in glibc's default), then tries the next server; do not edit /etc/resolv.conf on a real machine to prove it, use dig @192.0.2.1 as above. See dig and Why dig works but the app fails.
Lab 7: TLS certificate failures
Goal: two different verify errors, from hosts that exist for this purpose (badssl.com).
All three exit with code 60, so match on the message. The curl trailer text ("More details here") was dropped. To see the certificate itself with openssl:
The notAfter date in the past is the expiry. Do not "fix" it with curl -k; that turns verification off and so removes the very check you are studying. See curl and openssl and TLS and certificate errors.
Lab 8: packet loss and latency
Goal: make a fast link slow and lossy on purpose. Linux, root, illustrative. tc qdisc ... netem is documented in tc-netem(8). Apply it only to loopback or an interface on a box you own:
(Illustrative; 10 probes only roughly sample a 5% loss, so a given run can show 0%, 10% or 20%.) Because loopback carries both directions of the conversation, a 100 ms delay shows up as about 200 ms RTT; apply it to a real interface of a second container to model one side. delay 100ms 20ms is mean 100 ms with 20 ms jitter. For bandwidth limits use the tbf or netem rate options in the same man page.
macOS substitute. Dummynet. This replaces the active pf ruleset, so undo it right after (illustrative, needs root):
See dnctl(8) and pfctl(8). Apple's Network Link Conditioner (from the Additional Tools for Xcode download) does the same with a GUI profile, and is simpler for app testing. In a container, run tc inside it with docker run --cap-add NET_ADMIN; the container's network namespace is then yours alone (Docker run reference, runtime privilege and capabilities).
Lab 9: a PMTUD black hole
Goal: small packets work and large ones do not. Mechanism: MTU, MSS and fragmentation. Linux, root, illustrative, on two network namespaces you create:
Here the failing interface is the sender's own, so ping is told at once. A real black hole is when the small link is in the middle and its ICMP "fragmentation needed" message is filtered, so the sender never hears and just retransmits. To model that, put a third namespace in the middle with forwarding on and an iptables -A FORWARD -p icmp -j DROP; the large ping then hangs silently instead of failing. The -M do flag sets DF on Linux. On macOS the flag is -D, and this local variant did run:
Lab 10: port exhaustion
Goal: run out of ephemeral ports. Mechanism and fixes: Port exhaustion and TIME_WAIT. Linux, root, illustrative. Shrink the range so it is reachable in seconds, open connections to one destination, and read the error:
After roughly 51 connections in TIME_WAIT toward the same destination, further connects fail (curl reports Cannot assign requested address, EADDRNOTAVAIL). The default range varies by distribution and kernel; use the value you noted, not the one printed here. macOS uses net.inet.ip.portrange.first and .last (49152 to 65535 on this machine) and sudo sysctl -w the same way.
Lab 11: slow receiver and zero window
Goal: a receiver that never reads makes the sender stall. The sink accepts and sleeps; the flooder sends until send() would block.
The receiver's Recv-Q is full at 65536 bytes (its receive buffer) and nobody is reading it, so it advertises a window of 0. On loopback the kernel can also buffer on the send side, so the exact byte count where send() blocks depends on the OS and buffer sizes; read the Recv-Q column, not the number printed by the script. In a capture you would see win 0 from the receiver and then the sender's zero-window probes (illustrative, needs root):
The mechanism is Flow control and the receive window. On Linux read the same state with ss -tn 'sport = :9300'. Undo: stop both scripts (kill %1 %2).
Gotchas
tc or pf rule in place and the machine is just slow nowcurl -k is a way to read the TLS errorInterview questions
Show answer
Run a listener on a port, then add an iptables -A INPUT -p tcp --dport PORT -j DROP rule and curl it with --connect-timeout. The client hangs and tcpdump shows the SYN retransmitted with doubling gaps. Delete the rule with -D and add -j REJECT --reject-with tcp-reset and the same curl is refused in 0 ms. Doing it on loopback or in a namespace keeps it off real traffic.
Follow-ups:
- Why do clouds drop rather than reject? A silent drop reveals nothing to a scanner and costs the host no reply traffic.
- What does
REJECT --reject-with icmp-port-unreachablelook like to a TCP client? AlsoECONNREFUSED, since TCP maps the ICMP error to a refusal.
Show answer
Use tc with the netem qdisc: tc qdisc add dev eth0 root netem delay 100ms 20ms loss 5% adds 100 ms of delay with 20 ms jitter and 5% random loss, and tc qdisc del dev eth0 root removes it. Test with ping and curl -w. Do it on a container or namespace interface you own so it affects only the system under test.
Follow-ups:
- What is the tradeoff? It shapes outgoing traffic at one interface only, so for symmetric effects you need both ends, and loopback doubles the delay.
- How would you limit bandwidth?
tbfor therateoption, in the same man page.
Show answer
Inject faults at a layer the service cannot cheat around. At the kernel, tc netem (delay, jitter, loss, reorder, duplication, rate) on the test host or the container's namespace. At the proxy, a tool such as Toxiproxy sits between service and dependency and adds latency, drops or slow close per connection, which is easy to script in CI. Then assert on the behaviour you care about: timeouts fire at the configured value, retries back off and are bounded, and the service sheds load instead of queueing forever.
Follow-ups:
- Why not only unit tests with mocks? A mock cannot produce a half-open connection or a zero window; the kernel is the source of those symptoms.
- What about jitter? Real paths vary, so a fixed delay hides timeouts that only trip on the tail; set a jitter value and watch p99.
Show answer
Make the server close with an abortive close. In Python, set SO_LINGER to (1, 0) on the accepted socket and call close(); the kernel sends an RST instead of a FIN, and the client's next read fails with ECONNRESET. Without code, kill a nc -l listener while a client is still connected and write again. In a capture it is a bare RST on an established connection.
Follow-ups:
- How is that different from refused? Refused is an RST in answer to a SYN, before any connection exists.
Show answer
A connect timeout with no application timeout set: the kernel retransmits the SYN with doubling gaps and gives up after tcp_syn_retries attempts, about 127 seconds on Linux with the default of 6 on the classic schedule (about 131 seconds on recent kernels). Cause on the path: a drop rule, a dead host or a bad route. Always set an explicit connect timeout in the client so it fails in seconds.
Follow-ups:
- How do you tell a drop from a dead host? Capture on the server: if the SYN arrives and nothing is sent back, it is a drop or the process; if it never arrives, it is the path.
Fun facts
- The
.invalidtop-level domain is reserved so that it never resolves, which makes it a safe NXDOMAIN source (RFC 6761). 192.0.2.0/24,198.51.100.0/24and203.0.113.0/24are set aside for documentation and are not supposed to be routed, so a packet sent to one is a reliable silent black hole (RFC 5737).netemwas written by Stephen Hemminger for Linux, and its man page,tc-netem(8), also covers reordering, duplication, corruption and rate limiting.- On this macOS run, a rebind that failed with
Address already in usesucceeded as soon asSO_REUSEADDRwas set, with the TIME_WAIT entry still present.
Cheat sheet
| Failure | Make it with | You see | Undo |
|---|---|---|---|
| Refused | curl to a port with no listener | curl: (7), 0 ms; SYN then RST | none |
| Connect timeout | iptables -j DROP (macOS: pfctl block drop) | curl: (28), SYN retransmits | iptables -D, pfctl -d |
| Read timeout | server that accepts and sleeps | curl: (28), connect near 0 | kill the server |
| Reset | SO_LINGER (1, 0) then close | curl: (56), ECONNRESET | none |
| Address in use | two servers on one port; restart in TIME_WAIT | EADDRINUSE | kill the server |
| NXDOMAIN | dig nosuchname.invalid | status: NXDOMAIN | none |
| SERVFAIL | dig dnssec-failed.org | status: SERVFAIL | none |
| TLS | curl https://expired.badssl.com/ | curl: (60) | none |
| Loss and latency | tc qdisc add dev lo root netem ... (macOS: dnctl pipe) | ping RTT and loss | tc qdisc del, dnctl -q flush |
| MTU black hole | ip link set va mtu 1400 with ping -M do | message too long, or a silent hang | delete the namespaces |
| Zero window | receiver that never reads | win 0, send() blocks | kill both ends |