Error Messages to Cause

TL;DR

• Every error is one layer's verdict. The kernel, the resolver, the TLS library and the gateway each have their own vocabulary, so read the error as "who said no, and how".
• Refused, timeout, reset is the core trichotomy. Refused is an RST in answer to a SYN (a host replied: nothing listens). Timeout is silence (a drop, a dead host or a bad route). Reset is an RST on a connection that already existed.
• A connect timeout and a read timeout are different failures. The first means no SYN-ACK; the second means the connection is up and the peer did not answer. Fix them in different places.
• 502, 503 and 504 are written by a gateway, not your app. 502: the upstream refused, reset or sent garbage. 503: no healthy upstream or deliberate shedding. 504: the upstream did not answer in time.
• errno numbers differ between Linux and macOS (ECONNREFUSED is 111 on Linux, 61 on macOS). Match on the name, never the number.

Mental model

ANALOGYEveryday

You phone a company. A recorded voice saying "this number is not in service" is refused: somebody answered fast, and the answer was no. Ringing forever is a timeout: nobody picked up, and you cannot tell whether the office is closed or the line is cut. Being cut off mid-conversation with a click is a reset. And "the receptionist says the manager is not responding" is a 504: a real person told you, on behalf of someone who said nothing.

Where it breaks: a phone has one kind of silence. On a network the silence can be a firewall that deliberately drops, a router with no path, or a host that is off, and from the caller's side they are identical. That is why a timeout is the least informative error and the first to check from the other end.

ANALOGYFor engineers

Each error is an exception type thrown by a different layer of the call stack. ECONNREFUSED is the kernel's TCP code, EAI_NONAME is the C library's resolver, a certificate error is the TLS library's verifier, and a 502 is the reverse proxy's own handler. Reading them is the same skill as reading a stack trace: find the frame that threw, because that layer is the one that knows the cause.

Where it breaks: a stack trace names the throwing function exactly. A network error names only the layer, and several different causes (a firewall DROP, a dead host, a wrong route) throw the same exception, so you must confirm the cause with a second tool.

How it works

The three TCP verdicts

A TCP client's connect() starts with a SYN. What comes back, or does not, decides the error. A host with no listener on that port answers a SYN with a RST (RFC 9293 section 3.5.2, reset generation), and the client's connect() fails with ECONNREFUSED. If nothing answers at all, the kernel keeps retransmitting the SYN and eventually gives up with ETIMEDOUT. If the connection is already established and a RST arrives, the next read fails with ECONNRESET. The handshake and states are on TCP connection management and TCP state machine.

Left: a SYN is answered by a RST, so connect() fails fast with ECONNREFUSED. Middle: SYNs vanish (dashed with a cross) and the kernel retries with growing gaps until it gives up with ETIMEDOUT. Right: a connection that was working is cut by a RST, so the next recv() fails with ECONNRESET.

The key habit: a refusal proves the host is up and reachable (something sent a RST back along the whole path), while a timeout proves nothing except that no reply arrived. Who sends the RST is not always the server itself: a firewall rule can REJECT and send a RST (or an ICMP port unreachable, which Linux also turns into ECONNREFUSED) on the server's behalf, per iptables-extensions(8), so "refused" means "something with the right to speak for that address said no".

Where an HTTP error comes from

An HTTP status code is written by whichever hop gave up. When there is a CDN or load balancer in front of the app, the client talks to that hop, and the hop writes a 502, 503 or 504 when the upstream conversation went wrong. The client never sees the upstream's errno: it sees only the status line. The proxy's own log, however, has the errno.

The gateway turns three kinds of upstream failure into three different codes. The client sees only the status code; the gateway's own log holds the underlying errno. If the gateway itself is unreachable, the client sees a TCP error instead of any HTTP status.

The full status-code list is on HTTP status codes and semantics; this page owns the errno-to-cause mapping and the 502 versus 503 versus 504 diagnosis below.

Deep dive

Connect timeout versus read timeout

Both are reported as "timeout", and they mean opposite things.

• Connect timeout: the SYN got no SYN-ACK. The connection never existed. Suspects: a firewall or security group that drops, a wrong route, a dead host, a full SYN backlog. The kernel itself retries the SYN: Linux defaults to tcp_syn_retries = 6, which is 127 seconds on the classic schedule (1 second initial RTO, doubling) and about 131 seconds on recent kernels, where tcp_syn_linear_timeouts defaults to 4 (kernel ip-sysctl reference). Applications usually set a shorter timer, so they time out first.
• Read timeout: the connection is established, the request went out, and the peer did not answer in time. The network is fine. Suspects: a slow or hung application, a lock, a saturated backend. On an established connection the kernel can also give up retransmitting data with ETIMEDOUT; tcp_retries2 = 15 gives a lower bound of 924.6 seconds per the same reference.

The names differ per tool, and that is how you tell them apart:

Tool	Connect timeout	Read timeout
curl	exit 28, "Failed to connect ... Timeout was reached"	exit 28, "Operation timed out after N milliseconds with M bytes received"
Python	TimeoutError: timed out (raised from connect)	TimeoutError: timed out (raised from recv)
Go	dial tcp ...: i/o timeout	read tcp ...: i/o timeout
Java	SocketTimeoutException from Socket.connect	SocketTimeoutException from a read
Node	no timer by default: a connect that gets no reply waits for the kernel to give up with ETIMEDOUT unless you set socket.setTimeout	socket.setTimeout emits a timeout event, not an error; you must destroy the socket

curl exits 28 for both, so read the message, not the number. Python and Java use the same exception type for both, so look at the line that raised. Go's prefix (dial versus read) is the cheapest signal. An application-set timeout has no errno: Python's TimeoutError: timed out is a local timer firing, while a real ETIMEDOUT is the kernel giving up. See Java Socket and ConnectException: refusal throws ConnectException, a timer throws SocketTimeoutException. Timer design is on timeouts, retries and backoff.

What "refused" does and does not tell you

ECONNREFUSED means a RST came back to the SYN. Causes, in the order worth checking:

1. Nothing is listening on that port: the service is down, crashed, or has not finished starting.
2. Wrong port or wrong host: a typo, or a stale config pointing at the old port.
3. Listening on the wrong interface. A server bound to 127.0.0.1 refuses connections to the machine's LAN address, and a server bound only to IPv6 refuses an IPv4 connect. Node reports the address it tried, for example connect ECONNREFUSED ::1:3000, because localhost may resolve to ::1 first while the server listens on 127.0.0.1 only (Node dns.setDefaultResultOrder).
4. A REJECT firewall rule, which looks identical to "nothing listening". See firewalls.

First command: ss -ltnp on Linux, or lsof -nP -iTCP -sTCP:LISTEN on macOS (macOS has no ss), to see what is bound to which address. Then connect to that exact address.

Reset: the peer said abort

ECONNRESET ("Connection reset by peer") means a RST arrived on an established connection. The sender of the RST decided to abort, rather than close politely with a FIN. Usual causes:

• The server process crashed or was killed and the kernel reset its sockets.
• An idle-timeout mismatch. The load balancer or NAT drops an idle connection (an ALB's idle timeout is 60 seconds by default, per AWS: connection idle timeout), but the client's pool still believes the connection is open, so the next request on it gets a RST.
• A half-open connection after a reboot. One side restarted and lost the state; the survivor's next segment gets a RST.
• An abortive close. A close with unread data in the receive buffer sends a RST instead of a FIN, and so does SO_LINGER with a zero timeout.
• A middlebox with a connection-tracking table that expired or overflowed. See NAT.

EPIPE ("Broken pipe") is the write-side partner: writing to a socket that has already been reset or closed by the peer. On Linux a write to such a socket also raises SIGPIPE, whose default action kills the process unless the program ignores it or passes MSG_NOSIGNAL (send(2)). Which call sees the failure first, recv (ECONNRESET) or send (EPIPE), depends on timing and on the OS, so treat the two as the same event.

Bind errors: EADDRINUSE and EADDRNOTAVAIL

These come from the server side's bind() or from a client with no free source port, not from the network.

• EADDRINUSE (Linux 98, macOS 48): another socket holds that address and port, or a closing connection's TIME_WAIT does and the server did not set SO_REUSEADDR. Find the owner with ss -ltnp 'sport = :8080' or lsof -nP -iTCP:8080 -sTCP:LISTEN. The TIME_WAIT mechanics are on port exhaustion and TIME_WAIT.
• EADDRNOTAVAIL (Linux 99, macOS 49): two different meanings. Binding to an IP address that is not assigned to this host (a typo, or an interface that is down), or a client that has run out of ephemeral ports toward one destination. The second one is the production surprise and is fully explained, with the arithmetic, on port exhaustion and TIME_WAIT.

No route: EHOSTUNREACH and ENETUNREACH

• ENETUNREACH ("Network is unreachable"): the routing table has no route to the destination at all, often a missing default route or a down interface. The packet never left.
• EHOSTUNREACH ("No route to host"): a route exists but the host could not be reached. Typically an ICMP "destination unreachable" came back from a router (RFC 792), or the host is on your own segment and ARP got no answer, which shows as "Destination Host Unreachable" from the local host. See ARP and ICMP and ping.

Name resolution failures

The resolver has its own vocabulary, in two layers. The DNS server answers with a response code (RCODE, RFC 1035 section 4.1.1); the C library then maps the outcome to an EAI_* code from getaddrinfo, which is not an errno (getaddrinfo(3)).

What you see	Layer	Meaning
NXDOMAIN (RCODE 3)	DNS server	The name does not exist. An authoritative "no". Check for a typo or a missing record.
SERVFAIL (RCODE 2)	DNS server	The resolver could not produce an answer: unreachable or broken authoritative servers, a lame delegation, or a failed DNSSEC validation. A resolver failure, not a statement about the name.
REFUSED (RCODE 5)	DNS server	Policy: the server will not answer you, for example a recursive query to an authoritative-only server or an ACL.
EAI_NONAME: "Name or service not known" (glibc), "nodename nor servname provided, or not known" (macOS)	libc	The lookup completed and found no such name.
EAI_AGAIN: "Temporary failure in name resolution"	libc	A temporary failure: no resolver reachable, or it timed out. Retry may help.

The mechanics are on DNS and DNS records and routing (including DNSSEC), and the debugging commands are on dig and DNS debugging. The distinction to keep: NXDOMAIN is "the name is absent", SERVFAIL is "I could not find out". And dig talks to a resolver directly while an application goes through the C library, so a name that works in dig can still fail in the app (for example through /etc/hosts or nsswitch order).

TLS and certificate errors

A TLS failure has two parts: the verification error your library reports locally, and the alert the peer may send (the alert vocabulary is in RFC 8446 section 6.2 and on TLS). The OpenSSL verify error numbers are listed in OpenSSL's verification options.

Message	Usual cause	What to check
"certificate has expired" (OpenSSL error 10)	The leaf or an intermediate is past its notAfter, or the client clock is wrong	openssl s_client -connect host:443 -servername host and compare dates
"unable to get local issuer certificate" (error 20)	The server did not send the intermediate, or the client CA bundle is missing the root	Count the certificates the server sent; test from a second client
"hostname mismatch" or "no alternative certificate subject name matches"	The name you connected to is not in the certificate's SAN list	Print the SANs; check you used the right hostname and SNI
"self signed certificate" (18) or "self signed certificate in chain" (19)	A leaf signed by itself, or a private CA the client does not trust	Install the CA, or fix the chain
handshake_failure (alert 40), protocol_version (alert 70)	No common cipher suite or TLS version, for example a server with a minimum version the client does not support	openssl s_client -tls1_2 and -tls1_3 against it
certificate_unknown (alert 46)	The peer rejected the certificate for a reason it did not specify	Read the peer's logs

curl reports certificate trouble as exit code 60 ("peer certificate cannot be authenticated with known CA certificates") or 51 for a failed verification, and a handshake that cannot complete as 35 (curl exit codes). Which of 60 or 51 you get depends on the TLS backend and curl version: on the macOS curl in the Hands-on section below, all four certificate problems exit 60, including the hostname mismatch.

"Works in Chrome, fails in curl or Java" is the classic missing intermediate: some browsers can fetch a missing intermediate on their own, while OpenSSL-based clients do not, so only the strict clients fail. "Fails only on old devices" is usually an out-of-date CA store or a client that sends no SNI.

The 502 versus 503 versus 504 diagnosis

Three questions separate them:

1. Did the upstream answer? If it refused, reset or sent something that is not HTTP, the gateway writes 502. If it said nothing past the proxy's timeout, 504.
2. Did the gateway have anywhere to send it? If every target is unhealthy or none is registered, the gateway writes 503. An app that deliberately sheds load also answers 503, and may add Retry-After (RFC 9110 section 10.2.3).
3. Whose timeout fired first? A 504 is the gateway's timeout, not the app's. nginx's proxy_read_timeout and proxy_connect_timeout both default to 60 seconds (ngx_http_proxy_module), and an ALB's idle timeout is 60 seconds (ALB troubleshooting). If the gateway's limit is lower than the app's real worst case, healthy-but-slow requests become 504s.

Who logs it matters. An ALB access log has both elb_status_code (what the balancer returned) and target_status_code (what the target returned, - when the target gave no response) (ALB access logs); nginx exposes $status and $upstream_status (ngx_http_upstream_module). A 502 or 504 with an empty target status means the app never produced a status, and the app's own log will have no line for that request.

Related codes seen at the same layer:

• 499 is nginx's own, logged when the client closed the connection before the response; it is defined in nginx's source as NGX_HTTP_CLIENT_CLOSED_REQUEST and is in no RFC (nginx source).
• 429 is the rate limiter saying slow down, with an optional Retry-After; it comes from a policy, not a failure (429 and Retry-After).
• Cloudflare 520 to 526 are Cloudflare's own codes for an origin problem: 521 is a refused connection, 522 a TCP connect timeout, 524 a connected origin that did not respond in time, 525 a failed TLS handshake and 526 an invalid origin certificate (Cloudflare 5xx errors). They map cleanly onto the TCP errors above: 521 is "refused", 522 is "connect timeout", 524 is "read timeout".

Proxy behaviour in depth is on proxies and gateways and load balancing.

The rest of the family

• ECONNABORTED ("Software caused connection abort"; Linux 103, macOS 53): a connection was aborted locally by the host's software, and accept() can return it when a pending connection was aborted before it was accepted (accept(2)).
• EMFILE ("Too many open files", 24 on both): the process hit its file-descriptor limit. It is not a network failure, but a busy server sees it from accept() and socket(). Check ulimit -n.
• ENETDOWN ("Network is down"): the interface is down.
• EMSGSIZE ("Message too long"): a datagram is larger than the socket will send, for example a UDP datagram over the maximum size, or one larger than the path MTU with the Don't Fragment bit set (ip(7)). See MTU, MSS and fragmentation.
• Chrome's ERR_* names are not errno values. Chromium has its own error list, so the cheat-sheet tables below pair each Chrome name with the errno or code it corresponds to by meaning.

Textbook vs real world

The textbook says	Production does	Why
Connection refused means the server is down	It often means the server is up but listening on a different interface, port or address family	A RST proves a host answered, not that the service you wanted is the one that answered. A 127.0.0.1-only bind refuses LAN connections
A timeout means a firewall is dropping packets	It can equally be a dead host, a wrong route, a full SYN backlog or, after the connection is up, a slow application	Silence has many sources and the client cannot tell them apart. Read timeouts are not network failures at all
504 means the app is slow	Often the load balancer's timeout is lower than the app's legitimate worst case, or the proxy could not reuse an upstream connection	The 504 is the gateway's clock firing; the app may be healthy and still working
502 means the app crashed	Also a protocol mismatch (speaking plaintext HTTP to an upstream that expects TLS, or the reverse), a keep-alive connection the app closed, or an invalid response	The gateway writes 502 for any upstream reply it cannot use, not only for a dead process
Errors are portable across systems	errno numbers and message strings differ between Linux and macOS, and between libc versions	Each OS defines its own errno.h; only the names are common
A reset means the other end is broken	A reset is often an idle timeout, a middlebox or a stale pooled connection, with nothing wrong in either application	The sender chose to abort for a policy reason, and a middlebox can send a RST on behalf of an endpoint

Hands-on

All outputs below were captured on a macOS machine (curl 8.7.1, Python 3.14, Node 26, Go 1.27) and are abbreviated. Linux differs in errno numbers and a few message strings; the differences are listed after each block. Port 59871 had nothing listening. 192.0.2.1 is a documentation address (RFC 5737) that normally goes nowhere, so the timeout outcome depends on your network.

Refused, in four languages:

curl -sS http://127.0.0.1:59871/

python3 -c "import socket; socket.create_connection(('127.0.0.1', 59871))"

node -e "require('net').connect(59871,'127.0.0.1').on('error', e => console.log(e.code, e.message))"

curl: (7) Failed to connect to 127.0.0.1 port 59871 after 0 ms: Couldn't connect to server

ConnectionRefusedError: [Errno 61] Connection refused

ECONNREFUSED connect ECONNREFUSED 127.0.0.1:59871

A Go net.Dial printed dial tcp 127.0.0.1:59871: connect: connection refused. Same cause, four spellings. On Linux the Python errno is 111, not 61.

A connect timeout:

curl -sS --connect-timeout 2 http://192.0.2.1/

curl: (28) Failed to connect to 192.0.2.1 port 80 after 2002 ms: Timeout was reached

Go printed dial tcp 192.0.2.1:80: i/o timeout for the same target.

A read timeout. Run a server that accepts connections and never replies, then:

curl -sS --max-time 2 http://127.0.0.1:59874/

curl: (28) Operation timed out after 2004 milliseconds with 0 bytes received

Same exit code 28 as the connect timeout, different message. In Python recv raised TimeoutError: timed out, in Go the read printed read tcp 127.0.0.1:58714->127.0.0.1:59874: i/o timeout, and in Node the timeout event fired with no error object.

A reset and a broken pipe. A server that closes with SO_LINGER set to zero sends a RST instead of a FIN:

c.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))

c.close()

send: BrokenPipeError 32 [Errno 32] Broken pipe

recv: ConnectionResetError 54 [Errno 54] Connection reset by peer

Both are the same RST seen by two calls. On Linux the reset errno is 104, and which call reports first can differ.

Bind errors. Bind a second socket to a port that is already listening, then to an address this host does not own:

OSError 48 [Errno 48] Address already in use

OSError 49 [Errno 49] Can't assign requested address

macOS prints "Can't assign requested address" with an apostrophe; Linux prints "Cannot assign requested address" and errno 99. Find the owner of the first one with lsof (macOS) or ss (Linux):

lsof -nP -iTCP:59873 -sTCP:LISTEN

COMMAND PID USER FD TYPE NAME

Python 84569 abhishekr 3u IPv4 TCP 127.0.0.1:59873 (LISTEN)

On Linux: ss -ltnp 'sport = :59873'.

A DNS failure.

curl -sS http://no-such-host.invalid/

curl: (6) Could not resolve host: no-such-host.invalid

Python's socket.getaddrinfo raised gaierror: [Errno 8] nodename nor servname provided, or not known for the same name. Note that .invalid is reserved to never resolve (RFC 6761), so it is a safe name to test with. The Linux (glibc) message is "Name or service not known".

Certificate failures against the test hosts at badssl.com:

for h in expired wrong.host self-signed untrusted-root; do curl -sS https://$h.badssl.com/ ; done

curl: (60) SSL certificate problem: certificate has expired

curl: (60) SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'

curl: (60) SSL certificate problem: self signed certificate

curl: (60) SSL certificate problem: self signed certificate in chain

Each also printed a longer paragraph pointing at the curl documentation, omitted here. The -k flag silences all four, which is the wrong fix.

A message too long. Sending a 70,000 byte UDP datagram on macOS:

OSError 40 [Errno 40] Message too long

On Linux EMSGSIZE is 90. The practical commands for each error are in the cheat sheet below; run the failing request again from a second machine to separate a client problem from a server problem.

Gotchas

GOTCHAS6

Connection refused means the server is down

A RST proves the host is up and something answered. The usual cause is that the service you want is not listening on that exact address and port: a crash, a wrong port, a 127.0.0.1-only bind, or an IPv6-only bind. A firewall REJECT rule produces the same error. Run ss -ltnp or lsof -iTCP -sTCP:LISTEN on the server before blaming the process.

A timeout means a firewall DROP

Silence has several sources: a DROP rule or a missing security group rule, a wrong route, a dead host, a SYN that the server's queue could not accept, or, if the connection was already up, a slow application. The client cannot distinguish them. Use tcpdump on the server to see whether the SYN even arrives.

504 means the app is slow

A 504 is the gateway's timeout. If the load balancer allows 60 seconds and the app legitimately needs 90, the request never fails in the app, it fails at the gateway. Compare the gateway's timeout with the app's worst-case latency before profiling the app.

502 means the app crashed

A 502 means the gateway could not use the upstream's reply. That includes the app closing a connection the proxy was reusing, an invalid response, and a protocol mismatch such as plaintext to a TLS upstream. A crash is one cause among several; check the gateway's error log for the underlying errno.

curl exit 28 is always a network problem

Exit 28 is used for both the connect timeout and the read timeout. Read the message: "Failed to connect ... Timeout was reached" is a connect timeout, "Operation timed out after N milliseconds with M bytes received" is a read timeout, and a read timeout points at the server's application, not the network.

The errno number is portable

ECONNREFUSED is 111 on Linux and 61 on macOS; ETIMEDOUT is 110 and 60; ECONNRESET is 104 and 54. Code that compares against a literal number works on one OS and silently misses on the other. Always compare against the symbolic name.

Interview questions

INTERVIEWWhat is the difference between connection refused, a timeout and a reset?junior

Show answer

Refused means the SYN was answered with a RST: a host replied and nothing is listening on that address and port, so the failure is fast. A timeout means no answer at all: a firewall dropping, a dead host or a bad route, so the failure takes as long as the timer. A reset means a RST arrived on a connection that was already established: the peer crashed, aborted, or a middlebox cut it.

Follow-ups:

• Which one implies a firewall? A timeout, if the rule is a DROP. A REJECT rule looks like a refusal instead.
• Which one implies the process crashed? A refusal if it is down now, a reset if it was down mid-connection.
• How do you confirm from the server? tcpdump for the SYN: if it arrives with no SYN-ACK, the host or a local firewall is the problem; if it never arrives, look upstream.

INTERVIEWWhat is the difference between 502, 503 and 504?junior

Show answer

All three are written by a gateway. 502 means the upstream refused, reset or sent an invalid response. 503 means the gateway has no healthy upstream, or the app is deliberately shedding load, and it may carry Retry-After. 504 means the upstream did not answer within the gateway's timeout.

Follow-ups:

• Who retries? A client may retry 503 after the Retry-After delay; a 502 or 504 on a non-idempotent request is not safe to retry blindly. Proxies can retry idempotent requests on another upstream if configured.
• Which are the app's fault? None by definition, since the gateway wrote them, but the cause is usually the app or the gateway's timeout configuration.

INTERVIEWHow do you tell a connect timeout from a read timeout?mid

Show answer

A connect timeout means no SYN-ACK arrived, so the connection never existed: suspect a firewall, route or dead host. A read timeout means the connection is up and the peer did not respond in time: suspect a slow or hung application. The tool's message usually says which: curl says "Failed to connect ... Timeout" versus "Operation timed out after N ms with M bytes received", and Go prefixes dial versus read.

Follow-ups:

• Why do Python and Java look the same? Both use one exception type for both; the stack frame (connect versus read) tells you.
• Does an app timeout have an errno? No. It is a local timer. A real ETIMEDOUT is the kernel giving up retransmitting.

INTERVIEWUsers see intermittent 502s right after a deploy. What do you check?mid

Show answer

The classic cause is a keep-alive mismatch: the load balancer or proxy reuses an idle upstream connection that the app has just closed, and the first request on the dead connection fails with a reset, which the gateway turns into a 502. Deploys make it visible because old instances close connections. Check the proxy's error log for "connection reset by peer" or "upstream prematurely closed", and compare the app's keep-alive timeout with the load balancer's idle timeout.

Follow-ups:

• What is the fix? Make the app's keep-alive timeout longer than the balancer's idle timeout, so the balancer always closes first.
• What about deregistration? Drain connections before stopping an instance so in-flight requests finish.

INTERVIEWWhat is the difference between NXDOMAIN and SERVFAIL?mid

Show answer

NXDOMAIN (RCODE 3) is an authoritative answer that the name does not exist, so the record is absent or mistyped. SERVFAIL (RCODE 2) means the resolver could not obtain an answer: the authoritative servers are unreachable or broken, the delegation is lame, or a DNSSEC validation failed. NXDOMAIN is about the name; SERVFAIL is about the resolution process.

Follow-ups:

• What is REFUSED? The server declines to answer you at all, as a matter of policy.
• What is the application-side difference? EAI_NONAME is a completed lookup with no such name; EAI_AGAIN is a temporary failure worth retrying.

INTERVIEWA client making many outbound calls gets EADDRNOTAVAIL. What is happening?mid

Show answer

Ephemeral port exhaustion. Toward one destination IP and port, only the source port varies, so the client has roughly 28,000 ports on Linux, and every closed connection holds its tuple in TIME_WAIT for 60 seconds. At a high connection rate the pool empties and connect() fails immediately. The fix is fewer, longer-lived connections: pooling and keep-alive, then more source or destination addresses.

Follow-ups:

• Does tcp_tw_reuse fix it? It helps outgoing connections only, with caveats; pooling is the real fix.
• Where else does it hit? A NAT gateway shares the same port budget across everything behind it.

INTERVIEWUsers see ECONNRESET only under load. How do you investigate?senior

Show answer

Load changes which resource runs out first. Candidates: the server's accept queue overflowing, a connection-tracking table filling on a NAT or firewall, an idle or maximum-connections limit on a proxy, or file-descriptor exhaustion. Confirm with ss -ltn (on a listener, Recv-Q is the current accept queue and Send-Q the maximum backlog, per ss(8)), nstat -az TcpExtListenOverflows TcpExtListenDrops, the kernel log for nf_conntrack: table full, and tcpdump 'tcp[tcpflags] & tcp-rst != 0' to see who sends the RST.

Follow-ups:

• How do you tell the sender? The RST's source address and TTL in the capture: a RST whose TTL differs from the server's other packets was likely injected by a middlebox.
• What mitigations? Raise the backlog and somaxconn, size the tracking table, add capacity, and put limits on retries.

INTERVIEWA certificate error appears only on some clients. Why?senior

Show answer

Different clients differ in what they trust and what they will fetch. The common causes are a missing intermediate (the server sends only its leaf, and clients that do not fetch the issuer by themselves fail), an old CA store that lacks a newer root, and clients that send no SNI and so get a default certificate for the wrong name. A wrong client clock also looks like "expired".

Follow-ups:

• How do you check the chain? openssl s_client -connect host:443 -servername host -showcerts and count the certificates sent.
• Which client tells the truth? The strictest one, curl or OpenSSL: if it works there the server is sending a complete chain.

Fun facts

FACTSError trivia

• ECONNREFUSED is 111 on Linux and 61 on macOS, ETIMEDOUT is 110 and 60. The same name carries a different number on each OS, and even the message for ETIMEDOUT differs: Linux says "Connection timed out", macOS says "Operation timed out" (checked on macOS with Python's os.strerror).
• 499 appears in no RFC. It is nginx's private code for "the client closed the request before I answered", defined in its source as NGX_HTTP_CLIENT_CLOSED_REQUEST.
• A write to a closed socket can kill your process. On Linux, EPIPE also raises SIGPIPE, whose default action is to terminate the program, which is why network servers ignore it or pass MSG_NOSIGNAL (send(2)).
• Python's "timed out" has no errno. It is raised by a local timer, unlike a kernel ETIMEDOUT.
• The .invalid top-level domain is reserved so it can never resolve (RFC 6761), which makes it the safe name for testing NXDOMAIN-style failures.

Cheat sheet

NUMBERSNumbers to remember

• 127 s - Linux default total SYN retry time on the classic schedule (tcp_syn_retries = 6; about 131 s on recent kernels), after which connect() returns ETIMEDOUT
• 60 s - default ALB idle timeout, and nginx proxy_read_timeout and proxy_connect_timeout: the usual 504 clock
• 924.6 s - lower bound on data-retransmission give-up on an established connection (tcp_retries2 = 15)
• 111 / 110 / 104 - Linux ECONNREFUSED / ETIMEDOUT / ECONNRESET; macOS uses 61 / 60 / 54
• exit 28 - curl for both connect and read timeouts; 7 refused; 6 unresolved host; 35 / 51 / 60 TLS

Errors from the network (connect, send, receive)

Error as printed	Code	What happened on the wire	Usual cause	First command	Owner page
curl 7, ConnectionRefusedError, Node ECONNREFUSED, Go connection refused, Java ConnectException, Chrome ERR_CONNECTION_REFUSED	ECONNREFUSED	SYN answered by RST	Nothing listening, wrong port, listening on 127.0.0.1 or IPv6 only, REJECT rule	ss -ltnp (Linux), lsof -nP -iTCP -sTCP:LISTEN (macOS)	TCP connection, Firewalls
curl 28, TimeoutError, Go i/o timeout, Java SocketTimeoutException, Chrome ERR_CONNECTION_TIMED_OUT	ETIMEDOUT (kernel) or no errno (app timer)	No SYN-ACK (connect) or no response bytes (read)	DROP rule or missing security group, wrong route, dead host; for read timeouts a slow application	nc -zv -w 3 host port, tcpdump -nn host H and port P	Debugging playbook, Timeouts and retries
Connection reset by peer, curl 56, Chrome ERR_CONNECTION_RESET	ECONNRESET	RST on an established connection	Server crash, LB idle timeout below client keep-alive, half-open after reboot, abortive close, conntrack	tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0'	TCP states, NAT
Broken pipe	EPIPE	Write after the peer closed or reset	Same as reset; the write side of it	Look for the RST that came first	TCP states
Address already in use	EADDRINUSE	No packet: bind() failed locally	Another process, or TIME_WAIT without SO_REUSEADDR	ss -ltnp 'sport = :PORT', lsof -iTCP:PORT	Port exhaustion and TIME_WAIT
Cannot assign requested address	EADDRNOTAVAIL	No packet: bind() or connect() failed locally	IP not on this host, or out of ephemeral ports toward one destination	ip addr, ss -tan state time-wait | wc -l	Port exhaustion and TIME_WAIT
No route to host	EHOSTUNREACH	ICMP destination unreachable, or ARP got no answer	Host down on the local segment, filtered router	ip route get IP, ip neigh, traceroute IP	ARP, ICMP and traceroute
Network is unreachable	ENETUNREACH	No packet sent: no route in the table	Missing default route, interface down	ip route	Forwarding and routing
Software caused connection abort	ECONNABORTED	A connection was aborted before accept() returned it	Client gave up while queued	ss -ltn	TCP connection
Too many open files	EMFILE	None: a file-descriptor limit	Process fd limit, fd leak	ulimit -n, lsof -p PID	ss, netstat and curl
Network is down	ENETDOWN	None: the interface is down	Interface down	ip link	Debugging playbook
Message too long	EMSGSIZE	None: send refused locally	Datagram over the maximum, or over the path MTU with DF set	ping -M do -s SIZE host (Linux)	MTU, MSS and fragmentation

Name and certificate errors

Error as printed	Code	What happened	Usual cause	First command	Owner page
curl 6 Could not resolve host, Chrome ERR_NAME_NOT_RESOLVED	EAI_NONAME or EAI_AGAIN	The DNS lookup failed before any connection	Typo, missing record, resolver unreachable	dig name, dig @8.8.8.8 name	DNS records and routing, dig and DNS debugging
NXDOMAIN	RCODE 3	Authoritative: name does not exist	Missing or mistyped record	dig name +noall +comments	DNS
SERVFAIL	RCODE 2	Resolver could not get an answer	Broken authoritative servers, lame delegation, DNSSEC failure	dig name +trace, dig name +cd	DNS records and routing
REFUSED	RCODE 5	Server declined the query	Policy, recursion not allowed	dig @server name	DNS
curl 60 certificate has expired, Chrome ERR_CERT_DATE_INVALID	OpenSSL 10	Verification failed locally	Expired certificate or wrong client clock	openssl s_client -connect host:443 -servername host	TLS
unable to get local issuer certificate, Chrome ERR_CERT_AUTHORITY_INVALID	OpenSSL 20	Chain does not lead to a trusted root	Missing intermediate, old CA bundle	openssl s_client -showcerts	TLS
no alternative certificate subject name matches, Chrome ERR_CERT_COMMON_NAME_INVALID	OpenSSL hostname mismatch	Name not in the SAN list	Wrong hostname, missing SAN, no SNI	openssl s_client ... | openssl x509 -noout -ext subjectAltName	TLS
self signed certificate	OpenSSL 18 or 19	Chain ends at an untrusted root	Private CA not installed	openssl verify	TLS
curl 35 SSL connect error, handshake_failure, protocol_version	Alerts 40 and 70	Peer sent a fatal alert	No shared cipher suite or TLS version	openssl s_client -tls1_2	TLS

HTTP gateway errors

Status	Who writes it	What happened upstream	Usual cause	First command	Owner page
502 Bad Gateway	Proxy, LB, CDN	Upstream refused, reset, closed early or sent an invalid reply	App crash, keep-alive mismatch, protocol mismatch	Proxy error log; curl -v against the upstream directly	Proxies and gateways
503 Service Unavailable	LB, or the app	No healthy upstream, or deliberate shedding	All targets failing health checks, overload, maintenance	Target health view; check Retry-After	Load balancing
504 Gateway Timeout	Proxy, LB, CDN	Upstream silent past the gateway's timeout	Gateway timeout below app latency, hung backend	Compare gateway timeout with curl -w '%{time_total}' to the app	Timeouts and retries
499	nginx	Client closed first	Client timeout shorter than the server's	nginx access log	Proxies and gateways
429	Rate limiter	Policy rejected the request	Over quota	Read Retry-After	HTTP status codes
520 to 526	Cloudflare	Origin connection problem (521 refused, 522 connect timeout, 524 read timeout, 525 and 526 TLS)	Origin down, firewall blocking Cloudflare, origin certificate	Connect to the origin directly	HTTP status codes

Linux and macOS errno numbers

Name	Linux	macOS	Linux message	macOS message
ECONNREFUSED	111	61	Connection refused	Connection refused
ETIMEDOUT	110	60	Connection timed out	Operation timed out
ECONNRESET	104	54	Connection reset by peer	Connection reset by peer
EADDRINUSE	98	48	Address already in use	Address already in use
EADDRNOTAVAIL	99	49	Cannot assign requested address	Can't assign requested address
EHOSTUNREACH	113	65	No route to host	No route to host
ENETUNREACH	101	51	Network is unreachable	Network is unreachable
ENETDOWN	100	50	Network is down	Network is down
ECONNABORTED	103	53	Software caused connection abort	Software caused connection abort
EMSGSIZE	90	40	Message too long	Message too long
EPIPE	32	32	Broken pipe	Broken pipe
EMFILE	24	24	Too many open files	Too many open files

The macOS column was read from Python's errno and os.strerror on a macOS machine. The Linux column follows the kernel's errno.h and errno-base.h (the numbers above are the generic ones used on x86 and Arm); the names are listed in errno(3). Check on the machine you debug on: python3 -c "import errno; print(errno.ECONNREFUSED)".

Owner pages, linked: TCP connection, TCP state machine, port exhaustion and TIME_WAIT, firewalls, NAT, ARP, ICMP and ping, DNS, DNS records and routing, dig and DNS debugging, TLS, proxies and gateways, load balancing, timeouts, retries and backoff, HTTP status codes, debugging by layer, ss, nc, lsof and ip, failure lab.