VPNs & IPsec
TLS secures a single application's connection. Sometimes you want to secure all traffic between two networks or between a device and a network - as if they shared a private wire, even though the packets cross the public internet. That is what a VPN (Virtual Private Network) does, and IPsec is the protocol suite that most commonly powers it.
What a VPN does
A VPN builds an encrypted tunnel across the public internet. Traffic that would otherwise be exposed is wrapped in an encrypted outer packet, sent to the far end of the tunnel, unwrapped there, and delivered onward. To the two endpoints it feels like a direct private link; to anyone watching the public path it is opaque.
An observer on the public path sees only encrypted packets flowing between the two tunnel endpoints. They cannot read the inner traffic, the real destination addresses, or the ports. What they can still see is that the two tunnel endpoints are talking and roughly how much data flows. A VPN hides contents and internal addressing, not the existence of the tunnel itself.
IPsec
IPsec secures traffic at the network layer, so it protects entire IP packets regardless of the application on top. It offers two protocols:
- AH (Authentication Header) provides integrity and authentication for the packet, but no encryption - the payload stays readable.
- ESP (Encapsulating Security Payload) provides encryption plus integrity and authentication. Because it actually keeps data confidential, ESP is what real VPNs use.
Tunnel mode vs transport mode
IPsec can protect a packet in two ways, differing in how much of the original packet is protected and whether a new IP header is added:
| Transport mode | Tunnel mode | |
|---|---|---|
| What is protected | Only the payload of the IP packet | The entire original IP packet |
| Original IP header | Kept and visible | Encrypted inside a new outer packet |
| New outer IP header? | No | Yes - addressed to the tunnel endpoints |
| Typical use | Host-to-host on the same network | Site-to-site and remote-access VPNs |
Transport mode encrypts just the payload and leaves the original header in place - fine for two hosts talking directly. Tunnel mode encrypts the whole original packet and wraps it in a brand-new outer packet addressed to the tunnel endpoints. Because it hides the original addresses too, tunnel mode with ESP is the standard choice for VPNs.
Wrapping the original packet in a new outer packet is not free - ESP in tunnel mode typically adds somewhere around 50-73 bytes of overhead per packet: a new outer IP header (20 bytes for IPv4), an ESP header and initialization vector (roughly 8-16 bytes depending on cipher), padding to the cipher's block size, and an ESP trailer plus authentication data (roughly 12-16 bytes). Take a client sending a full Ethernet-sized packet at the standard 1500-byte MTU: add ~60 bytes of IPsec overhead and the packet is now ~1560 bytes - too big for any link still capped at 1500. Every network interface in the tunnel path now needs an MTU that accounts for that overhead (commonly the tunnel interface is set to something like 1400 or 1436 bytes), or the sender must fragment before encrypting.
A classic IPsec failure mode: the tunnel establishes fine, small packets (DNS queries, TCP handshakes) sail through, but anything larger - a big HTTPS response, a file transfer - just hangs or times out. The cause is almost always the overhead above: the encrypted, encapsulated packet now exceeds the path's real MTU, but ICMP "fragmentation needed" messages often cannot get back to the sender because a firewall along the path blocks ICMP, or because the packet was marked Don't Fragment and something between the endpoints silently drops it instead of signaling the problem. The result looks exactly like the general PMTUD black hole (see MTU, MSS & Fragmentation), except it is now hidden a layer deeper, inside a tunnel most tools do not introspect. The standard fix is to clamp the effective MSS on the tunnel interface (an "MSS clamp") so TCP itself negotiates a smaller segment size that leaves room for the IPsec overhead, rather than relying on fragmentation and PMTUD to save the day.