ss, nc, lsof and ip - Sockets and Interfaces
ss(Linux only) dumps the kernel's socket table:ss -ltnpanswers "who is listening and which process owns it",ss -tananswers "who is connected". On macOS uselsof -iTCP -sTCP:LISTEN -n -Pandnetstat -an -p tcp; on Windowsnetstat -ano.Recv-QandSend-Qmean different things on a LISTEN row (accept queue length, and its maximum) than on a connected row (bytes not yet read by the app, bytes not yet ACKed by the peer).nc -zv host 443tests whether a TCP handshake completes. Refused means a reset came back (host up, nothing listening). Timed out means silence (dropped, or no route). Neither says the application is healthy.- The address a socket is bound to matters:
127.0.0.1is reachable only from the host itself,0.0.0.0or*from every IPv4 address on it. ip route get 1.1.1.1shows which interface, gateway and source address the kernel would pick.ip neighis the ARP/NDP table,ip -s linkthe per-interface drop and error counters.ipis Linux; macOS usesifconfig,route,arp.
Mental model
A building directory in the lobby. One column lists which rooms have someone waiting to take visitors (the listening ports), another lists who is on a call with whom right now (the connections), and a note beside each says which tenant owns the room. You are not watching the corridors, only reading the directory.
Where it breaks: the directory shows what the building's front desk (the kernel) has recorded, not what the tenants think. A tenant can have hung up mentally while the desk still lists the call, and a room can be listed as open while a security guard (a firewall) turns every visitor away.
The kernel's socket table is a queryable in-memory database. Each row is a socket: protocol, local and remote address, state, owner, queue sizes, timers. ss is SELECT ... WHERE state = ... AND dport = ... against that table, and it reads it through a netlink socket instead of parsing text files. /proc/net/tcp is the same table as a raw dump, and lsof is the join from sockets to file descriptors to processes.
Where it breaks: a database row is the truth. A socket row is only the kernel's bookkeeping: ESTABLISHED can outlive a dead peer, and LISTEN says nothing about whether the program behind it answers requests.
How it works
Which tool on which OS
ss is part of Linux's iproute2 package. It does not exist on macOS or Windows, and "command not found" on a Mac is the usual first symptom. Pick the row for your OS, not the tool you remember.
| Question | Linux | macOS | Windows |
|---|---|---|---|
| Who is listening, and which process? | ss -ltnp (needs root to see other users' processes) | lsof -iTCP -sTCP:LISTEN -n -P | netstat -ano, then tasklist /fi "pid eq N"; or Get-NetTCPConnection -State Listen |
| Who owns port 8080? | ss -ltnp 'sport = :8080' | lsof -iTCP:8080 -n -P | Get-NetTCPConnection -LocalPort 8080 |
| All connections | ss -tan | netstat -an -p tcp | netstat -an |
| Is a remote port open? | nc -zv host 443 | nc -zv host 443 | Test-NetConnection host -Port 443 |
| Interfaces and addresses | ip -br addr | ifconfig | ipconfig |
| Routing table | ip route | netstat -rn | route print |
| Which route for this IP? | ip route get 1.1.1.1 | route -n get 1.1.1.1 | Find-NetRoute -RemoteIPAddress 1.1.1.1 |
| Neighbour (ARP) table | ip neigh | arp -an | arp -a |
The macOS commands in this table were run on the macOS box used for the captures below. The Linux commands follow ss(8), ip(8) and ip-route(8). The Windows commands are standard Microsoft tools and were not captured here, so no Windows output is shown on this page.
Reading ss
ss takes flags that combine like ls flags. These are the ones that carry the work (ss(8)):
| Flag | Meaning |
|---|---|
-t / -u | TCP sockets / UDP sockets |
-l | Only listening sockets (by default ss hides them) |
-a | All sockets, listening and not |
-n | Numeric addresses and ports: no DNS or service-name lookups, so it is faster and does not mislead you with names |
-p | Show the owning process and PID. Only your own processes appear unless you are root |
-o | Show timers (retransmit, keepalive, TIME-WAIT countdown) |
-s | Summary counts per state, not a row per socket |
-i | Per-socket TCP internals: rtt, cwnd, retransmissions |
-e | Extended info: user id and inode |
Filters go at the end, so you ask the kernel instead of piping through grep: ss -tn state established, ss -tn dst 10.0.0.5, ss -tn 'dport = :443', ss -ltn 'sport = :8080'. The filter grammar is in ss(8).
The rows below are illustrative Linux output (addresses from the documentation ranges), because ss is not on the macOS box used for the real captures:
The -i line is the one interviewers like. rtt:3.2/1.1 is smoothed RTT and its variance in ms, cwnd:10 is the congestion window in segments, retrans:0/2 is retransmissions in flight and in total. What those fields mean is owned by TCP performance and flow control.
Worked example: who owns port 8080
Your server fails to start with "address already in use". Something holds the port.
- Linux:
ss -ltnp 'sport = :8080'prints the LISTEN row. Theusers:(("python3",pid=3320,fd=3))column names the process. Without root, a process owned by another user shows an empty last column, which is the "no -p output" gotcha. - macOS:
lsof -iTCP:8080 -n -P. Real capture from this Mac, with a throwawaypython3 -m http.server 8080 --bind 127.0.0.1running:
Read it left to right: the process (Python, PID 24821), its user, the file descriptor it holds the socket on (3u, opened read-write), the address family, and finally the bound address. -n stops lsof resolving host names and -P stops it turning port 8080 into http-alt. Then kill 24821 or fix the config. If the owner is gone but the bind still fails, look for TIME_WAIT instead: see port exhaustion and TIME_WAIT.
Deep dive
Recv-Q and Send-Q: two meanings
The two numeric columns are the most-misread part of the output. The full explanation, including a wording trap in the ss(8) man page, is owned by the TCP state machine page. The reading rule to carry around:
| Row | Recv-Q | Send-Q |
|---|---|---|
| LISTEN | Connections that finished the handshake and wait for the app to call accept() (accept queue length now) | Accept queue capacity: min(listen() backlog, net.core.somaxconn) |
| ESTAB | Bytes received that the app has not read yet | Bytes sent that the peer has not yet acknowledged |
A non-zero value on its own means little. Persistent growth is the signal: Recv-Q on ESTAB climbing means the application is not reading, Send-Q climbing means the peer or the path is not draining, Recv-Q close to Send-Q on LISTEN means the app is not calling accept() fast enough.
The two queues are a Linux design described in listen(2): since Linux 2.2 the backlog argument sizes the queue of completed connections, the incomplete-connection queue is sized separately by tcp_max_syn_backlog, and the backlog is silently capped at net.core.somaxconn. That file defaults to 4096 since Linux 5.4 and was 128 before (same man page). Cloudflare's SYN packet handling in the wild walks through the two queues and their failure modes.
Accept-queue overflow: the idle-CPU timeout
A classic SRE case: clients see connect timeouts under load while the server CPU is idle. If the application is slow to call accept() (blocked threads, GC pause, a single-threaded loop), the accept queue fills and the kernel drops new handshake completions without sending a reset, so clients only see a retransmitting SYN. The evidence is on the host:
A rising TcpExtListenOverflows (nstat(8)) plus a LISTEN row whose Recv-Q sits near its Send-Q confirms it. The fixes are a faster accept() loop first, then a larger listen() backlog together with net.core.somaxconn; raising one without the other changes nothing because the smaller wins. The kernel behaviour is detailed on the TCP state machine page.
Reading states from the tooling side
The meaning of each state, and the close sequence that produces it, is owned by the TCP state machine. From the tooling side three patterns matter:
- CLOSE-WAIT growing: the peer closed and your application never called
close(). It is a descriptor leak in your process, not a network fault. Find the owner withss -tanp state close-wait. - SYN-SENT stuck: your side sent SYNs and nothing came back. A firewall or security group is dropping them, or there is no route. An active refusal would have returned a reset at once.
- Many TIME-WAIT: the side that closed first is holding the four-tuple. Usually harmless; it only hurts when it exhausts ports, which is the subject of port exhaustion and TIME_WAIT.
ss -sgives the totals quickly.
nc: a probe and a listener
nc (netcat) opens a raw TCP or UDP socket and connects stdin and stdout to it. For debugging it is two commands: nc -zv host 443 (-z scan without sending data, -v report) tests whether a handshake completes, and nc -l 8080 makes a listener you can talk to. Real captures from this Mac:
The three outcomes are three different network facts. Succeeded means the SYN got a SYN-ACK: something accepts connections. Refused means a RST came back: the host is reachable and no process listens (or a firewall rejects actively). Timed out means silence: a firewall dropping, a missing route or a dead host. The address 192.0.2.1 is TEST-NET-1 (RFC 5737), reserved for documentation, so nothing answers it. Add -u for UDP, but note a UDP "success" only means no ICMP port-unreachable came back, since UDP has no handshake.
nc is not one program. The macOS nc is OpenBSD-derived (nc -l 9100 works, and the usage line lists -z). The OpenBSD man page is nc(1). The older traditional/GNU netcat needs nc -l -p 9100, and Nmap's ncat has its own option set; on Debian-family systems nc can point at either the openbsd or traditional build. If a flag is rejected, check nc -h and the man page on that host before assuming a typo.
ip: addresses, routes, neighbours, counters
ip (iproute2, Linux) replaces ifconfig, route and arp (ip(8), ip-route(8)). The cheat sheet:
| Command | Answers |
|---|---|
ip -br addr | One line per interface: state and addresses (-br is brief) |
ip link | Interface state, MTU, MAC address |
ip -s link | Same plus RX/TX packets, errors and drops per interface |
ip route | The routing table |
ip route get 1.1.1.1 | Which route, gateway, interface and source address the kernel would use for that destination |
ip neigh | The ARP (IPv4) and NDP (IPv6) neighbour table with states such as REACHABLE and STALE |
ip route get is the fastest way to settle "which path does this packet take": it runs the real lookup instead of making you read the table. The longest-prefix and default-route logic behind it is owned by the host forwarding decision, and what a neighbour entry is and why it can be stale is owned by ARP. The macOS counterparts, captured here, are in Hands-on.
/proc/net/tcp: when there is no ss
In a minimal container without iproute2, the same table is readable as text from /proc/net/tcp (and /proc/net/tcp6). Addresses are hex and the state is a hex code (0A is LISTEN), so it is a last resort, but it works with nothing installed (see proc(5)).
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
netstat lists connections and listeners | Use ss on Linux; netstat is still the tool on macOS and Windows | netstat belongs to the older net-tools package, deprecated on Linux, and ss reads kernel socket info through netlink instead of parsing /proc, which matters on hosts with many sockets. macOS and Windows never had ss |
| A port that shows LISTEN is reachable | LISTEN says only that a process called listen(); reachability is a separate question | A firewall, security group, or a bind to 127.0.0.1 can still stop every outside client. Test from the client side with nc -zv |
nc -zv success means the service works | It means a handshake completed, nothing more | The kernel completes the handshake before the app reads a byte, so a hung or crashing app still accepts into its queue until the queue fills. Use a protocol-level check for health |
In production
- Finding leaked sockets and descriptors. A growing CLOSE-WAIT count pinned to one PID is the standard signature of a connection leak;
ss -tanp state close-waitnames the process andls /proc/PID/fd | wc -lshows the descriptor count against itsulimit -n. The mechanism is on the TCP state machine page. - Listener queue tuning. Servers such as nginx expose the
backlogparameter of thelistendirective, and the effective queue is still capped bynet.core.somaxconn; Cloudflare describes how SYN and accept queues behave under load in SYN packet handling in the wild. - Nodes with huge TIME-WAIT counts.
ss -sis the first command to run on a busy node; the budget behind the number is explained in port exhaustion and TIME_WAIT. - Systematic debugging. These tools are layer 3 and 4 checks in the sequence laid out on debugging by layer.
History and evolution
- 1983
netstatships with 4.2BSD, the BSD release that popularised the sockets API, as the way to list sockets and routes. - Mid-1990s
netcatappears as a "TCP/IP Swiss army knife" for reading and writing raw connections. OpenBSD and Nmap later ship their own incompatible rewrites, which is whyncflags differ today. - 2000s on Linux, iproute2 (
ip,ss) arrives to replace net-tools (ifconfig,route,arp,netstat).ssuses netlink, so it avoids parsing/proctext and stays fast with many sockets. net-tools is now deprecated on Linux. - 2019 Linux 5.4 raises the default
net.core.somaxconnfrom 128 to 4096 (listen(2)), because a 128-deep accept queue overflowed on busy servers. - Now
ssis the standard Linux tool; macOS still usesnetstatandlsof, Windows usesnetstatand PowerShell cmdlets.ss -Kcan forcibly close sockets where the kernel supports it (ss(8)). - Next eBPF tools (bpftrace, bcc) trace socket events as they happen rather than polling a snapshot.
Hands-on
All output below is from this macOS box (macOS 27.0.1), abridged. ss and ip are not available here, so Linux rows earlier on the page are labelled illustrative.
Who is listening, with owners (names of other users' processes are hidden unless you run as root):
*:49296 is the wildcard bind (any address); 127.0.0.1:49166 is loopback only. Different processes, different exposure.
The connection table, and the macOS listen queue view:
macOS netstat writes address.port with a dot, not a colon. netstat -Lan is the macOS analogue of the LISTEN row: qlen/incqlen/maxqlen is accept queue length, incomplete (half-open) queue length and the maximum. Python's http.server asked for a backlog of 5 here, which is why maxqlen is 5.
TIME_WAIT after a quick nc test. I ran nc -l 127.0.0.1 9100 and connected to it with nc 127.0.0.1 9100; both exited at once. Afterwards netstat -an -p tcp | grep 9100 showed both ends of the closed connection still held:
Interfaces, routes and the neighbour table (macOS equivalents of ip addr, ip route get, ip neigh):
route -n get 1.1.1.1 is the macOS form of ip route get 1.1.1.1: the destination matched the default route, via gateway 192.168.0.1 out of en0. The arp -an line is the gateway's MAC, which is the neighbour entry the forwarding step needs (ARP). The ifconfig mask 0xffffff00 is a /24. Windows equivalents (netstat -ano, Get-NetTCPConnection, Test-NetConnection) are in the table under How it works; their output was not captured here.
Gotchas
Interview questions
Show answer
On Linux, ss -ltnp 'sport = :8080' lists the listening socket and its owning process; on macOS, lsof -iTCP:8080 -n -P does the same. Both show the PID, which you can then inspect or kill. Use -n so you are not waiting on name lookups.
Follow-ups:
- Why do I need root? Process ownership of other users' sockets is hidden from unprivileged users, so
-porlsofshows only your own. - What is the macOS command?
lsof -iTCP:8080 -n -P;ssdoes not exist there.
Show answer
nc -zv host 443 attempts a TCP handshake and reports the result without sending data. Success means a SYN-ACK came back. It proves reachability at layer 4 and nothing about the application above it.
Follow-ups:
- What do refused and timed out mean? Refused is a RST: the host is up and nothing is listening, or a firewall rejects actively. Timed out is silence: a drop, a missing route or a dead host.
- Does it work for UDP?
-usends a probe, but with no handshake you only learn that no ICMP unreachable came back.
Show answer
On a connected socket, Recv-Q is bytes received but not yet read by the application and Send-Q is bytes sent but not yet acknowledged. On a LISTEN socket, Recv-Q is how many completed connections wait in the accept queue and Send-Q is that queue's maximum, min(backlog, somaxconn). Persistent growth is the signal, not a single non-zero reading.
Follow-ups:
- What does a large Recv-Q on ESTAB point at? The application is not reading fast enough.
- And a large Send-Q? The peer or path is not draining, or the peer's window is closed.
Show answer
Yours, the local application's. CLOSE-WAIT means the peer sent FIN, the kernel ACKed it and is waiting for your program to call close(). Nothing in the protocol times it out, so a leak grows until the process runs out of descriptors. Find the owning process with ss -tanp state close-wait.
Follow-ups:
- How is that different from TIME-WAIT? TIME-WAIT is the side that closed first holding the tuple briefly and is normal; CLOSE-WAIT is an unfinished close that is a bug.
Show answer
Either the old process is still alive and holding the listener (find it with ss -ltnp or lsof -iTCP:PORT), or connections from it are in TIME-WAIT. A listening socket bound without SO_REUSEADDR can fail to bind while old connections on that port linger; setting SO_REUSEADDR allows the bind. The distinctions are on port exhaustion and TIME_WAIT.
Follow-ups:
- Does
SO_REUSEADDRlet two live listeners share a port? No, only a quick re-bind past lingering connections;SO_REUSEPORTis the one for multiple listeners.
Show answer
Suspect accept-queue overflow. Run ss -ltn on the listening port: Recv-Q near Send-Q means the app is not accepting fast enough. Confirm with nstat -az TcpExtListenOverflows rising. The kernel drops the handshake without a reset, so clients simply retransmit SYNs and time out. Fix the accept loop first, then raise both the listen() backlog and net.core.somaxconn.
Follow-ups:
- Why raise both? The effective queue is the smaller of the two.
- Why is CPU idle? The app is blocked or starved, not busy; the bottleneck is how often it reaches
accept().
Show answer
ss -ti, optionally with a filter such as dst 203.0.113.9. It prints TCP internals under each socket: smoothed RTT and variance, cwnd, ssthresh, MSS and retransmission counts. It reflects what the kernel believes now, so repeat it during the problem. Interpretation of those fields is on TCP performance.
Follow-ups:
- Is it on macOS? No,
ssis Linux; macOS exposes aggregate TCP counters throughnetstat -s -p tcp.
Fun facts
ss -Kcan forcibly close sockets when the kernel supports it (ss(8)), so the tool that reads the socket table can also edit it.- The default Linux
net.core.somaxconnwas 128 for years and became 4096 in Linux 5.4 (listen(2)). - On this Mac,
python3 -m http.serverasked for a listen backlog of 5, visible asmaxqlen5 innetstat -Lan. - 192.0.2.1 is reserved for documentation (RFC 5737), which makes it a handy guaranteed-silent target for demonstrating a
nctimeout.
Cheat sheet
- Who listens, and which process?
ss -ltnp(Linux),lsof -iTCP -sTCP:LISTEN -n -P(macOS),netstat -ano(Windows). - Is a port reachable from here?
nc -zv host port. Refused is a reset, timed out is silence. - Where does a packet go?
ip route get IP(Linux),route -n get IP(macOS). - Is the accept queue overflowing?
ss -ltnRecv-Q against Send-Q, thennstat -az TcpExtListenOverflows. - Who is stuck in CLOSE-WAIT?
ss -tanp state close-wait. - Per-connection RTT and cwnd?
ss -ti.
curl has moved
curl -v, the -w timing fields and --resolve now live on curl and openssl, next to openssl s_client.