Skip to main content

Port Exhaustion and TIME_WAIT in Production

TL;DR
  • A connection is identified by a 4-tuple. Toward one server, three fields are fixed and only the source port varies, so a client has about 28,232 ports (Linux default) per destination.
  • The side that closes first parks the 4-tuple in TIME_WAIT: 60 seconds on Linux, hard-coded. 28,232 ports / 60 s is about 470 new connections per second to one IP:port before connect() fails.
  • The symptom is EADDRNOTAVAIL ("Cannot assign requested address") or connect timeouts at a NAT. It is never ECONNREFUSED.
  • SO_REUSEADDR does not recycle TIME_WAIT, tcp_tw_recycle was removed in Linux 4.12 because it broke NAT clients, and tcp_tw_reuse only helps outgoing connections.
  • The real fix is fewer, longer-lived connections: pooling and keep-alive. Cloud NAT gateways have the same budget, shared by everything behind the IP.

Mental model​

ANALOGYEveryday

A hotel on one street has 28,232 rooms. Every guest who checks out leaves the room held for 60 seconds before it can be rented again, in case a late courier still arrives with a parcel for the old guest. Send the hotel 1,000 new guests every second and it runs out of rooms after about 28 seconds; the front desk then turns people away even though most rooms are empty, just waiting out their hold.

Where it breaks: a client has one "hotel" per destination, so rooms toward a different server are a separate pool. And the hold is not a courtesy of the front desk: it is TCP protecting both sides from stray late packets.

ANALOGYFor engineers

It is exhausting a connection pool, except the pool is the set of source ports and the "return to pool" delay is 60 seconds after close. The pool key is the whole 4-tuple, so the pool is per destination, like a database pool configured per host. Opening a connection per request without reusing it is the same bug as creating a thread per request: it works until the rate crosses pool size / hold time.

Where it breaks: a thread pool blocks when empty. An exhausted port pool fails immediately with an error, or silently drops your SYNs if the exhaustion is on a NAT in front of you.

How it works​

The budget is the source port​

A TCP connection is identified by (source IP, source port, destination IP, destination port). The full story of why the kernel keys on all four is in Ports, Sockets and Demultiplexing. What matters here: when your client calls a single API at 198.51.100.7:443 from a single address, the first, third and fourth fields never change. Only the source port can differ between two concurrent connections, so the source-port range is your connection budget.

Source IPSource portDestination IPDestination port10.0.1.5kernel picks198.51.100.7443fixedthe only free fieldfixedfixed28,232 ephemeral ports032768609996553528,232 ports / 60 s TIME_WAIT = about 470 new connections per second
Toward one destination, three of the four fields are fixed. The Linux default ephemeral range 32768 to 60999 holds 60999 - 32768 + 1 = 28,232 ports. Dividing by the 60 second TIME_WAIT hold gives the sustainable rate of new connections.

The Linux default range is 32768 60999, documented under ip_local_port_range in ip(7) and the kernel ip-sysctl reference. Other systems pick differently: IANA's suggested dynamic range in RFC 6335 is 49152 to 65535 (16,384 ports), which is what macOS and modern Windows use by default.

Why a closed connection still holds its tuple​

When a connection closes, the side that sent the first FIN (the active closer) ends in TIME_WAIT and keeps the 4-tuple reserved for 2 * MSL before the state is released. The close sequence itself is on TCP Connection; the state definitions are in TCP State Machine. What the wait is for is two things. RFC 9293 section 3.6.1 requires the linger; the reasons are the standard textbook ones:

  1. The final ACK may have been lost. The peer is still in LAST_ACK and will retransmit its FIN. Someone has to be around to re-ACK it.
  2. Old duplicates must die. A delayed segment from the old connection must not be accepted as data on a new connection that reuses the same 4-tuple.
Client (active closer)Server (passive closer)FINACKFINfinal ACK (lost)FIN again after RTOACK, wait restartsFIN_WAIT_1FIN_WAIT_2TIME_WAITCLOSEDCLOSE_WAITLAST_ACKCLOSED60 s of quiet: no more FINs, tuple released
The client closes first, so it holds TIME_WAIT. The dashed final ACK is lost; the server's retransmitted FIN arrives during TIME_WAIT and is re-ACKed, which restarts the wait. After 60 seconds of quiet the tuple is released.

On Linux the hold is the compile-time constant TCP_TIMEWAIT_LEN, 60 seconds, in include/net/tcp.h. It is not a sysctl. The textbook value is 2 * MSL with MSL suggested as 2 minutes in RFC 793, which would be 4 minutes. Real stacks use shorter values:

WhereTIME_WAIT holdTunable?
Linux60 sNo, compiled in (TCP_TIMEWAIT_LEN)
macOS30 s (2 * net.inet.tcp.msl, default 15 s)Yes, sysctl net.inet.tcp.msl
FreeBSD60 s (2 * net.inet.tcp.msl, default 30 s)Yes, sysctl net.inet.tcp.msl
WindowsRegistry TcpTimedWaitDelay, default 120 s, range 30 to 300 s (2 to 300 s on Windows 8.1 and Server 2012 R2 and later)Yes, registry (the value does not exist by default)

Sources: the macOS default is TCPTV_MSL (15 seconds) plus the msl sysctl in Apple's open-source XNU kernel (tcp_timer.h, tcp_timer.c); FreeBSD defines TCPTV_MSL as 30000 ms (tcp_timer.h); Microsoft documents the Windows value under Adjust the MaxUserPort and TcpTimedWaitDelay settings. Apple publishes no user-facing page for msl, so confirm with sysctl net.inet.tcp.msl on your own build.

Worked example: one client, one API, no keep-alive​

A service calls https://api.example.com (one IP, port 443) at 1,000 requests per second, opening a new connection per request and closing it after the response. The client closes first, so every request leaves one TIME_WAIT.

StepCalculationResult
Ports available toward api.example.com:44360999 - 32768 + 128,232
Connections per second we trygiven1,000
Seconds until every port is in use (nothing has expired yet)28,232 / 1,000about 28 s
TIME_WAIT sockets we would need in steady state1,000 per s x 60 s60,000
Sustainable rate with this range28,232 / 60about 470 per s
Same, range widened to 1024-65535 (64,512 ports)64,512 / 60about 1,075 per s

After about 28 seconds connect() starts failing with EADDRNOTAVAIL, and it keeps failing for more than half of all attempts because only about 470 ports per second come free. Switch to a pool of 50 persistent connections and the same 1,000 requests per second produce zero new TIME_WAIT sockets.

Try it yourself: the starting preset is this exact example, so scrub past 28 seconds to watch the range run out, then turn on keep-alive, a second destination or a second source IP.

healthynear limit (over 80% of ports held)exhausted (connect() failing)
Load
Kernel settings

Linux default range 32768-60999 (28,232 ports), TIME_WAIT 60 s hard-coded; Windows holds 120 s by default.

healthy
Ports per (source IP, destination) pair, 1 pair: 28,232 in the range
■ in use: 100▨ TIME_WAIT: 1,000□ free: 27,132failed connects/s: 0
TIME_WAIT sockets per pair vs time
0 s60 s120 s180 s240 s28,2320
TIME_WAIT sockets (solid)port range (dashed)now (s)

Steady-state rule: new connections per second x TIME_WAIT must stay under the ports available per destination. Here 1,000 per s x 60 s = 60,000 held, against 28,232 ports (28,232 x 1 source IP). The range sustains about 471 new connections per second per destination. First failure at t = 28 s.

Healthy: ports return to the pool as fast as new connections take them.
second 0 / 240

What it looks like when it breaks​

Where the ports run outWhat your code sees
On the client hostconnect() fails immediately with EADDRNOTAVAIL, text "Cannot assign requested address" (connect(2) documents this for an unbound socket when every ephemeral port is in use)
On a NAT in front of youNew SYNs are dropped, so connect() hangs and times out; the NAT's own metric counts the drops
In the conntrack table of a Linux NAT or firewallKernel log nf_conntrack: table full, dropping packet, then the same timeouts
Nothing is listening at the destinationECONNREFUSED: an RST came back. This is not exhaustion

Deep dive​

Who owns the TIME_WAIT​

Only the active closer enters it. That decides where the pile builds up:

  • Client closes first (typical for clients that finish a request and hang up): each TIME_WAIT consumes one of the client's source ports toward that server. This is where exhaustion bites.
  • Server closes first (HTTP/1.0 style, Connection: close, idle timeouts, a server that kills connections after N requests): the TIME_WAIT sockets sit on the server. They do not eat the server's listening port, because their 4-tuples all differ by client address and port. The cost is kernel memory for the sockets, and net.ipv4.tcp_max_tw_buckets caps how many Linux keeps; past the cap, new ones are destroyed immediately and a warning is logged (ip-sysctl).

So "lots of TIME_WAIT on a server" is normally harmless. "Lots of TIME_WAIT on a client toward one destination" is the incident.

The knobs, and what each one really does​

KnobWhat it really doesSide effect or limit
Connection pooling / keep-aliveReuses open connections, so no new tuple and no TIME_WAIT per requestNeeds idle timeouts and health checks on pooled connections; this is the fix, not a workaround
ip_local_port_rangeWidens the pool, for example to 1024 65535 (64,512 ports)Buys about 2.3x headroom; takes ports away from anything that listens in that range
More source IPsMultiplies the budget: the tuple differs per source addressNeeds the addresses and routing; on a cloud NAT this means more NAT IPs
IP_BIND_ADDRESS_NO_PORTBinds a chosen source IP without reserving a port, so connect() picks a port unique only to the full 4-tuple (Linux 4.2+)Without it, bind-then-connect pins one port to one socket and the budget stops being per destination
tcp_tw_reuseLets connect() reuse a TIME_WAIT tuple when timestamps prove it is safeOutgoing connections only; needs tcp_timestamps; see below
SO_REUSEADDRLets bind() succeed on an address that has lingering TIME_WAIT socketsDoes not release TIME_WAIT or add ports; it is a server-restart fix
SO_REUSEPORTLets several sockets bind the same addr:port; the kernel spreads incoming connections across themA listener scaling tool; unrelated to TIME_WAIT
tcp_max_tw_bucketsCaps TIME_WAIT sockets; extras are destroyed at onceThrows away the protection TIME_WAIT gives; a DoS guard, not a tuning knob
tcp_tw_recycleWas a per-host timestamp shortcutRemoved in Linux 4.12; broke clients behind NAT

SO_REUSEADDR vs SO_REUSEPORT vs tcp_tw_reuse​

Three options with near-identical names, three different jobs:

  • SO_REUSEADDR (socket option, socket(7)): relaxes bind(). A server that restarts can bind its port again while old connections to it sit in TIME_WAIT. It also lets a specific-address bind coexist with a wildcard bind. It never shortens TIME_WAIT and never helps a client that is out of source ports.
  • SO_REUSEPORT (socket option, Linux 3.9): several sockets, typically one per worker, may bind the same address and port, and the kernel distributes incoming connections or datagrams among them. All must set it and run as the same user. It is about spreading load across listeners (LWN: The SO_REUSEPORT socket option). BSD and macOS use the same name with different semantics, so do not assume behaviour carries over.
  • tcp_tw_reuse (sysctl, ip-sysctl): the only one of the three that touches the TIME_WAIT problem. When connect() needs a port and the only candidate is a TIME_WAIT tuple, the kernel may reuse it if TCP timestamps (PAWS, RFC 7323) show the new connection's timestamps are newer than the old one's and about a second has passed. It applies to outgoing connections only. Documented values are 0 (off), 1 (global) and 2 (loopback only); recent kernels default to 2, so check your own kernel's documentation before assuming it does anything on a production path. The residual risks are analysed in RFC 6191 and, for the underlying hazard, RFC 1337.

Why tcp_tw_recycle was removed​

tcp_tw_recycle shrank the TIME_WAIT hold by remembering, per remote IP address, the most recent TCP timestamp seen, and dropping any SYN from that address whose timestamp was lower. Timestamps are per-host clocks. Put ten hosts behind one NAT: they share one public IP but have ten unrelated clocks, so the server sees timestamps jump backwards and silently drops SYNs from whichever hosts happen to be behind. Connections from mobile users, offices and corporate proxies failed intermittently with no log line to explain it. Vincent Bernat's Coping with the TCP TIME-WAIT state on busy Linux servers is the standard write-up, and the option was removed outright in Linux 4.12 (2017). If an interviewer says "just enable tw_recycle", the correct reply is that it no longer exists and why it was wrong.

Cloud NAT is the same budget, shared​

A NAT gateway rewrites many private sources onto a few public IPs (NAT explains the mechanics; NAT gateways and egress covers the cloud products). The translated flow still needs a unique (public IP, public port, destination IP, destination port, protocol) tuple, so the same arithmetic applies, but now every workload behind the IP spends from one pool.

VM AVM BVM CNAT gatewayDestination 1Destination 2ports 1024-1087ports 1088-1151ports 1152-1215public IP 203.0.113.10usable ports 1024-65535= 64,512 per IPper (dst IP, dst port, proto)198.51.100.7:443198.51.100.8:443One VM: at most 64 simultaneous connections to ONE destination.A different destination is a different tuple, so the same ports can be reused toward it.
Three VMs share one public IP. In this GCP-style static allocation each VM owns a slice of 64 ports, so one VM can hold at most 64 simultaneous connections to a single destination. A different destination is a different tuple, so the same ports may be reused toward it. Slice boundaries are illustrative.

What each provider documents, and where to read it:

  • AWS NAT gateway: up to 55,000 simultaneous connections to each unique destination (destination IP, destination port and protocol) per NAT gateway IP address; a different destination gets another 55,000. Failed source-port allocations increment the ErrorPortAllocation CloudWatch metric (NAT gateway basics, CloudWatch metrics, troubleshooting). AWS's own guidance for fixing it includes more IP addresses on the gateway, spreading across several gateways, and reducing connection churn.
  • GCP Cloud NAT: each NAT IP offers 64,512 ports (1024 to 65535) and ports are allocated per VM, with a configurable minimum per VM (the documented default is 64 for static allocation, the Public NAT default, and 32 for dynamic allocation). A VM that needs more simultaneous connections to one destination than its allocation fails; dynamic port allocation can grow it (ports and addresses).
  • Azure: a load balancer frontend public IP makes 64,000 ports eligible for SNAT, preallocated across backend instances (outbound connections). Azure NAT Gateway provides 64,512 SNAT ports per public IP, up to 16 IPs, allocated on demand across the subnet's VMs (SNAT with NAT gateway). Its SNATConnectionCount metric (filter ConnectionState for failed) and PacketDropCount metric show exhaustion (NAT gateway metrics); for load balancers see troubleshooting.

The NAT can also forget, or hold too long​

Pooling cures exhaustion but creates a new duty. A NAT or firewall expires idle flows, so a pooled connection that sat unused can be silently dead when you reuse it. AWS documents a 350 second idle timeout on NAT gateway flows (CloudWatch metrics, troubleshooting), so keep pooled connections alive with TCP keepalive or application pings well inside it, or recycle them proactively.

On the other side, a Linux NAT keeps a conntrack entry for a closed TCP flow in its own TIME_WAIT state, by default nf_conntrack_tcp_timeout_time_wait = 120 seconds (nf_conntrack sysctl docs). That is longer than the endpoints' 60 seconds, so a NAT box can run out of table space or ports before the hosts do.

Textbook vs real world​

The textbook saysProduction doesWhy
TIME_WAIT lasts 2 * MSL, which is 4 minutes with MSL = 2 minutesLinux holds it for a fixed 60 s; macOS 30 s, FreeBSD 60 s and Windows 120 s, all tunableRFC 793's MSL was a conservative guess; stacks shortened it, and Linux made it a constant
A host can have at most 65,535 connectionsThe limit is per 4-tuple: a client has about 28,232 per destination, a server can hold far more because clients varyPort numbers are 16-bit, but the tuple has three other fields
Set SO_REUSEADDR or tcp_tw_recycle and TIME_WAIT goes awaySO_REUSEADDR only fixes bind(); tcp_tw_recycle was removed in Linux 4.12Neither shortens the hold on a client that is out of source ports
Many TIME_WAIT sockets mean something is brokenOn a server that closes first it is routine; on a client toward one destination it is an incidentWhich side closes, and toward how many distinct destinations, decides
NAT hides the port problem from applicationsNAT is where the problem moves: every workload behind one IP spends one shared per-destination poolThe translated flow still needs a unique tuple
Tune the kernel until it worksReuse connections first; tune only after thatEach reused connection also skips a handshake, a TLS setup and slow start

In production​

  • Cloudflare documented running out of ephemeral ports on its own edge and the options for stretching the effective range, including IP_BIND_ADDRESS_NO_PORT and sharing a source port across destinations: How to stop running out of ephemeral ports and start to love long-lived connections. The conclusion matches this page: long-lived connections beat clever tuning.
  • AWS publishes the 55,000-connection per-destination limit and the ErrorPortAllocation metric, plus the guidance to add IPs or reduce churn (NAT gateway troubleshooting). Alarm on a non-zero sum of that metric.
  • Google Cloud documents per-VM port allocation and the exhaustion behaviour of Cloud NAT (ports and addresses).
  • Go's net/http is a classic cause: the default Transport keeps only 2 idle connections per host (DefaultMaxIdleConnsPerHost), so a service that fans out hundreds of concurrent requests to one backend closes most connections after each use and fills the client with TIME_WAIT. Raise MaxIdleConnsPerHost and always read and close response bodies so connections return to the pool (net/http Transport).
  • nginx needs keepalive in the upstream block, plus proxy_http_version 1.1; and an empty Connection header, to reuse upstream connections; without them each proxied request opens a new upstream connection (ngx_http_upstream_module).
  • I did not find primary-source engineering posts from Lyft or Netflix on this topic and have left them out rather than cite them from memory.

History and evolution​

HISTORY
  • 1981 RFC 793 specifies TIME_WAIT: after an active close, wait 2 * MSL (MSL suggested as 2 minutes) so a lost final ACK can be recovered and old segments expire. Connections were rare and long, so a 4-minute hold cost nothing.
  • 1992 RFC 1323 adds timestamps and PAWS (now RFC 7323), and RFC 1337 catalogues the "TIME_WAIT assassination" hazards of cutting the wait short.
  • 1990s and 2000s Web servers open and close a connection per request. HTTP/1.0 closes after every response; persistent connections in HTTP/1.1 and client-side pooling arrive in part to stop the churn.
  • 2011 RFC 6191 describes safely reducing TIME_WAIT using timestamps; Linux's tcp_tw_reuse follows that idea for outgoing connections.
  • 2013 and 2015 Linux 3.9 adds SO_REUSEPORT for multi-listener servers, and Linux 4.2 adds IP_BIND_ADDRESS_NO_PORT so clients can bind a source IP without burning a port.
  • 2014 tcp_tw_recycle is shown to silently drop SYNs from clients behind NAT (Bernat); advice flips from "enable it" to "never enable it".
  • 2017 Linux 4.12 removes tcp_tw_recycle entirely.
  • Now Port exhaustion is mostly a cloud NAT problem: a fleet shares a few public IPs, each cloud documents its per-destination limit, and the standard fixes are pooling, more IPs and private endpoints that bypass the NAT. HTTP/2 and HTTP/3 multiplex many requests on one connection, which removes the churn at its source.
  • Next QUIC runs over UDP and has no TIME_WAIT, but a NAT still holds a port mapping per UDP flow, so connection reuse stays the lever.

Hands-on​

Commands work on Linux unless marked macOS. Output is abbreviated and the numbers are illustrative.

Count and break down TIME_WAIT sockets:

ss -Htan state time-wait | wc -l
ss -Htan state time-wait | awk '{print $4}' | sort | uniq -c | sort -rn | head -5
ss -s
28187
28001 198.51.100.7:443
104 198.51.100.9:5432
TCP: 28435 (estab 212, closed 28190, orphaned 3, timewait 28187)

The second command groups TIME_WAIT sockets by peer address and port, which is the per-destination view that matters. One peer holding about 28,000 is the exhaustion signature (28,232 is the whole budget toward it). -H suppresses the header line so the count is exact.

Check the budget and the knobs:

sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse net.ipv4.tcp_timestamps
net.ipv4.ip_local_port_range = 32768 60999
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_timestamps = 1

Kernel counters and NAT state:

nstat -az TcpExtTW TcpExtTCPTimeWaitOverflow
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
conntrack -S | head -3

TcpExtTW counts sockets that ended their TIME_WAIT; TcpExtTCPTimeWaitOverflow rising means tcp_max_tw_buckets is destroying sockets early. On a NAT or firewall host, a conntrack count near the max, or growing drop and insert_failed in conntrack -S, means the table is the bottleneck.

Watch it climb. curl closes after each run, so every iteration normally leaves a client-side TIME_WAIT (-o /dev/null hides the body):

for i in $(seq 1 200); do curl -s -o /dev/null http://localhost:8080/ ; done
ss -Htan state time-wait '( dport = :8080 )' | wc -l
200

Now reuse one connection: curl's URL globbing sends 200 requests over one persistent connection.

curl -s -o /dev/null "http://localhost:8080/?[1-200]"

The TIME_WAIT count rises by one, not 200. That difference is the whole page.

On macOS use netstat -an -p tcp | grep -c TIME_WAIT, sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last and sysctl net.inet.tcp.msl; ss is not available. On AWS, check the NAT gateway for exhaustion:

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
--metric-name ErrorPortAllocation \
--dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
--start-time 2026-10-10T00:00:00Z --end-time 2026-10-10T06:00:00Z \
--period 300 --statistics Sum

Any non-zero Sum means connections failed for lack of ports in that window. The ss tool itself is covered in ss, nc, lsof and ip.

Gotchas​

GOTCHAS5
SO_REUSEADDR is not a TIME_WAIT recycler
You cannot shorten TIME_WAIT on Linux
Port exhaustion is per destination, not per host
'Connection refused' is not exhaustion
TIME_WAIT on a server is usually fine

Interview questions​

INTERVIEWWhy does TIME_WAIT exist, and why 2 * MSL?mid
Show answer

The side that closes first waits in TIME_WAIT for two reasons from RFC 9293: the final ACK might be lost, so the peer will retransmit its FIN and someone must re-ACK it, and delayed segments from the old connection must expire before the same 4-tuple is reused. 2 * MSL covers a segment's maximum lifetime in each direction: one MSL for the last packet to die in flight, one for a possible reply. The textbook MSL is 2 minutes, but Linux hard-codes the wait at 60 seconds.

Follow-ups:

  • What if the final ACK is lost? The peer in LAST_ACK retransmits its FIN, the TIME_WAIT side re-ACKs and restarts the timer.
  • Which side holds it? Whichever sent the first FIN.
INTERVIEWWe see "Cannot assign requested address" when calling a service at a high rate. Diagnose it.senior
Show answer

That is EADDRNOTAVAIL from connect(): the client has no free source port for that destination. Check ss -Htan state time-wait grouped by peer: one destination holding roughly 28,000 sockets confirms it. The cause is almost always a new connection per request with the client closing first, so each leaves a 60 second TIME_WAIT. The fix is connection pooling or keep-alive; widening ip_local_port_range or adding source IPs only raises the ceiling.

Follow-ups:

  • Is it also exhaustion if there is a NAT? Then it shows as timeouts instead, and you check the gateway metric.
  • Another cause of the same error? Binding to an address the host does not own.
  • Why per destination? The tuple's other three fields differ, so ports can be reused toward other servers.
INTERVIEWA server has tens of thousands of TIME_WAIT sockets. Is that a problem, and what do you do?mid
Show answer

Usually no. If the server closes first, TIME_WAIT lands on it, but those tuples differ by client so no port is exhausted, and the cost is small kernel memory. It becomes a concern if the count is hitting tcp_max_tw_buckets, or if the same pile is on a client toward one backend. The fixes are to let the client close first, use keep-alive and pooling so connections are reused, and avoid per-request Connection: close.

Follow-ups:

  • Can I lower the 60 s? Not on Linux; it is a compile-time constant.
  • What about tcp_max_tw_buckets? It discards TIME_WAIT sockets early, trading away protection against stray segments.
INTERVIEWExplain SO_REUSEADDR, SO_REUSEPORT and tcp_tw_reuse.senior
Show answer

SO_REUSEADDR lets bind() succeed when old connections on that port are in TIME_WAIT; it matters for server restarts and does not shorten TIME_WAIT. SO_REUSEPORT lets many sockets bind the same address and port and spreads incoming connections across them, which scales a listener across cores. tcp_tw_reuse is a sysctl that lets an outgoing connect() reuse a TIME_WAIT tuple when timestamps show it is safe; it is the only one that eases client-side exhaustion.

Follow-ups:

  • Does tcp_tw_reuse help servers? No, outgoing connections only, and it needs tcp_timestamps.
  • What is the default? Recent kernels use 2 (loopback only); read your kernel's ip-sysctl doc.
  • Is SO_REUSEPORT portable? The name exists on BSD and macOS but the semantics differ.
INTERVIEWWhy was tcp_tw_recycle removed?senior
Show answer

It tried to cut TIME_WAIT short by tracking the last TCP timestamp per remote IP and rejecting SYNs with older timestamps. Timestamps are per-host clocks, so clients sharing one public IP behind a NAT looked like one host whose clock jumped backwards, and their SYNs were silently dropped. It was removed in Linux 4.12 in 2017 and should never be recommended; tcp_tw_reuse carries the safe part of the idea.

Follow-ups:

  • Why did it appear to work in tests? Single clients and no NAT have monotonic timestamps.
  • What replaced it? Connection reuse, and tcp_tw_reuse for outgoing connections.
INTERVIEWExplain cloud NAT port exhaustion and how you fix it.senior
Show answer

A NAT gateway maps many private sources onto a few public IPs, and each mapping needs a unique (public IP, port, destination IP, destination port, protocol) tuple. So the budget is per public IP per destination: AWS documents 55,000 simultaneous connections per unique destination per gateway IP, and GCP allocates ports per VM. Workloads that open a connection per request to one external API burn that pool, and new connections time out. Fixes: pool and reuse connections, add NAT IPs or gateways, raise ports per VM, and use private endpoints so traffic skips the NAT.

Follow-ups:

  • How do you see it? AWS ErrorPortAllocation, GCP NAT dropped-packet metrics, Azure NAT Gateway SNATConnectionCount (failed state) and PacketDropCount.
  • Why not just add IPs forever? Cost, allow-lists on the destination side, and it leaves the churn in place.
INTERVIEWHow many connections can a client make to one server?mid
Show answer

About 28,232 simultaneous ones by default on Linux, the size of the ephemeral range 32768 to 60999, because the source port is the only free field. Widen the range to 1024 to 65535 and it is 64,512, near the 16-bit ceiling. Extra source IPs, or a second destination IP or port, each give another full budget. The server end is not limited this way: it holds one listening port and distinguishes connections by client address and port.

Follow-ups:

  • Sustained rate? Divide by the 60 s TIME_WAIT: about 470 per second with the default range.
  • Can a server hold more than 65,535 connections? Yes, limited by file descriptors and memory, not ports.
INTERVIEWWho ends up in TIME_WAIT, the client or the server?junior
Show answer

Whoever closes first, not whoever opened. If the client sends the first FIN it holds TIME_WAIT; if the server closes first it does. A server that drops idle keep-alive connections puts it on the server, while a client that hangs up after each request puts it on the client, which is the dangerous case.

Follow-ups:

  • Why is client-side worse? It consumes the client's scarce source ports toward one destination.
  • Both closing at once? Both sides go FIN_WAIT_1, CLOSING, TIME_WAIT.

Fun facts​

FACTSTIME_WAIT trivia
  • Linux's 60 seconds is a constant, TCP_TIMEWAIT_LEN, and tcp_fin_timeout happens to default to 60 as well, which is the source of much confusion.
  • RFC 793's suggested MSL of 2 minutes would make a TIME_WAIT 4 minutes, which is where the old "2 to 4 minutes" folklore comes from.
  • tcp_tw_recycle is one of very few Linux sysctls deleted outright because it could not be made safe (Linux 4.12).
  • Go's http.Transport defaulting to 2 idle connections per host is a well-known source of accidental TIME_WAIT floods.

Cheat sheet​

NUMBERSNumbers to remember
  • 28,232 - ports in the Linux default ephemeral range 32768 to 60999
  • 60 s - Linux TIME_WAIT, a constant (macOS 30 s via msl)
  • about 470 per s - sustainable new connections to one IP:port with the default range
  • 64,512 and about 1,075 per s - ports in 1024 65535 and the rate they support
  • 55,000 - AWS NAT gateway simultaneous connections per unique destination per IP
  • 64 - GCP Cloud NAT documented static-allocation default minimum ports per VM
  • 120 s - Windows TcpTimedWaitDelay default (Microsoft documents it as 0x78)
  • 120 s - Linux conntrack's default for closed TCP flows in its TIME_WAIT state
  • Linux 4.12 - tcp_tw_recycle removed
DECISIONOut of source ports: what do I do?
Client opens a connection per request to one backend→Pool and keep-alive; fix the client first
Pool exists but is tiny (for example 2 idle per host)→Raise the idle-connection limit and drain response bodies
Many workloads share a cloud NAT IP→Add NAT IPs or ports per VM, or use private endpoints
Still short after pooling→Widen ip_local_port_range, add source IPs, use IP_BIND_ADDRESS_NO_PORT
Server restart says address already in use→Set SO_REUSEADDR on the listener
Someone suggests tcp_tw_recycle→It was removed in Linux 4.12 and broke NAT clients
Pick this when: you can say which side closes first and toward how many destinations