Kubernetes Networking - Pods, Services, kube-proxy, CNI, Ingress
- The Kubernetes network model is flat: every pod gets its own IP, and pods reach each other across nodes without NAT (cluster networking). A CNI plugin is what builds that.
- Pod IPs are ephemeral. A Service gives a set of pods one stable virtual IP and a DNS name; clients use the name, never a pod IP.
- kube-proxy does not carry traffic. It programs kernel rules (iptables, IPVS or nftables) that DNAT the Service IP to one pod IP; conntrack remembers the choice so replies are un-translated (Virtual IPs and Service Proxies).
- Inside a pod,
ndots:5makes a name likeapi.example.comtry several cluster search domains before the real lookup. That is the usual cause of slow DNS in a cluster. - NetworkPolicy is default-allow. A pod no policy selects accepts everything, and policies only take effect if the CNI plugin enforces them.
Mental model
Picture an office where every desk has its own direct phone line (the pod IP). Anyone can ring any desk without going through a switchboard. Desks come and go as people change shifts, so you do not memorise numbers. Instead you ring the front desk (the Service), which has one fixed number and forwards your call to whoever is on shift. A directory (DNS) tells you the front desk's number by department name.
Where it breaks: the front desk here is not a person answering the phone. It is a set of forwarding rules installed in every building's phone system, so the call is redirected before it leaves your own floor. And a front desk forwards each new call to a randomly chosen person; it does not look at who is busy.
It is service discovery plus a DNAT load balancer, built into the kernel of every node. A Service is a name that resolves to a virtual IP, like a DNS-fronted VIP on a hardware load balancer, except no box owns the VIP. kube-proxy is the control loop that watches the API and rewrites the NAT rules, much as a config reloader rewrites an nginx upstream list. The mechanics of DNAT and conntrack are owned by the NAT page; the proxy and mesh concepts are on proxies and gateways.
Where it breaks: a normal load balancer is a process you can log into and watch. A ClusterIP lives only in rule tables. In iptables mode no interface holds the address, so there is nothing to ping and no process to attach a debugger to.
How it works
Kubernetes sets three rules for the network, and everything else follows from them (cluster networking):
- Every pod gets its own IP, taken from the node's slice of the pod CIDR.
- Pods can reach all other pods, on any node, without NAT: the destination sees the real pod IP as the source.
- A Service gets a virtual IP from a separate Service CIDR. Nothing is ever assigned that IP on a wire; it exists only as a match in packet-rewrite rules.
Inside a node, each pod lives in its own network namespace with a veth pair: one end in the pod, the other end in the node. The node then routes by its table exactly as described in the host forwarding decision. Between nodes, the CNI plugin decides how a pod IP is carried: an encapsulating overlay, plain routes, or the cloud's own network (see the deep dive).
Clients call the Service by name. CoreDNS answers orders.default.svc.cluster.local with the ClusterIP. The first packet leaves the pod addressed to that ClusterIP and, on the node's way in, kube-proxy's rules rewrite the destination to one of the Service's endpoints.
Worked example
Client pod 10.244.1.7 calls the Service orders (ClusterIP 10.96.12.5, port 80). It has two endpoints: 10.244.2.9:8080 and 10.244.3.4:8080. All addresses on this page are examples.
- The pod's resolver asks CoreDNS for
orders.default.svc.cluster.localand gets10.96.12.5. - The pod opens TCP to
10.96.12.5:80from ephemeral port 48310. The SYN reaches the node's netfilter hooks. - kube-proxy's rules match destination
10.96.12.5:80and pick one endpoint, here10.244.2.9:8080, at random. The destination is rewritten and conntrack records the pairing. - The packet is routed to the pod
10.244.2.9by whatever the CNI built. The source is still10.244.1.7, so the server sees the real client pod. - The server replies to
10.244.1.7:48310. When the reply reaches the client's node, conntrack rewrites the source back to10.96.12.5:80. The client never learns a pod answered.
Every later packet of that connection hits the conntrack row and skips the rule walk. The random choice is made once per connection.
The rules, abbreviated
In iptables mode the rewrite is a chain walk in the nat table: KUBE-SERVICES matches the Service, jumps to a KUBE-SVC-... chain, which picks one KUBE-SEP-... chain per endpoint with the statistic module, and the SEP chain does the DNAT (virtual IPs docs). Illustrative and abbreviated (chain suffixes are made up):
With two endpoints the first rule fires half the time and the second catches the rest. With three, the first has probability one third and the second one half of what remains. The rule list grows with the number of endpoints, which is why large clusters look at IPVS or nftables.
Service types
| Type | Reach | Port | Use | Cloud object created |
|---|---|---|---|---|
| ClusterIP | Inside the cluster only | Any port you choose on the virtual IP | Default. East-west traffic between workloads | None |
| NodePort | Any node IP from outside | One port per Service, default range 30000-32767 | Building block for load balancers; dev and bare metal | None |
| LoadBalancer | From outside via a cloud load balancer | The Service port on the balancer, forwarded to a NodePort or directly to pods | Expose one Service publicly | A cloud load balancer, provisioned by the cloud controller |
Headless (clusterIP: None) | Inside the cluster | Pod ports directly | DNS returns pod IPs themselves: StatefulSets, client-side balancing | None, and no kube-proxy rules |
| ExternalName | Inside the cluster | n/a | A DNS CNAME to an outside name | None |
Source: Service. A LoadBalancer Service builds on a NodePort, which builds on a ClusterIP. What each cloud provisions for it is the subject of cloud load balancers.
Endpoints come from EndpointSlices
A Service selects pods by label. The control plane tracks the matching, ready pods in EndpointSlice objects, and kube-proxy on every node watches them to rebuild its rules. EndpointSlices replaced the single Endpoints object so that a large Service does not have to be re-sent in full each time one pod changes; a slice defaults to at most 100 endpoints (EndpointSlices). A pod that fails its readiness probe is removed from the slice and stops receiving new connections.
Deep dive
How a pod IP crosses nodes: CNI families
The Container Network Interface is a small spec: the runtime calls a plugin binary with ADD or DEL and a network namespace, and the plugin wires an interface and returns an IP (CNI spec). Kubernetes does not care how the plugin carries traffic between nodes, only that the network rules hold.
| Family | Examples | Pod IP source | Encapsulation | MTU cost | NetworkPolicy |
|---|---|---|---|---|---|
| Overlay | Flannel (VXLAN), Calico in VXLAN or IP-in-IP mode, Cilium tunnel mode | Per-node slice of a pod CIDR you choose | Yes: pod packets wrapped in an outer UDP or IP packet between nodes | VXLAN costs 50 bytes, so 1450 inside a 1500 underlay; IP-in-IP costs 20 | Depends on the plugin: Calico and Cilium enforce, Flannel alone does not |
| Routed | Calico with BGP, no encapsulation | Per-node slice of a pod CIDR | None: nodes advertise pod routes (BGP) or the fabric learns them | None | Calico enforces |
| VPC-native | AWS VPC CNI, GKE VPC-native (alias IPs), Azure CNI | The cloud network itself: pod IPs are real VPC or VNet addresses | None: the cloud fabric routes pod IPs | None | Provided by the cloud plugin or an add-on; check the version |
| eBPF dataplane | Cilium, GKE Dataplane V2 | Per-node slice or cloud addresses, depending on mode | Optional | Same as the mode chosen | Cilium enforces, in eBPF |
Sources: Calico VXLAN and IP-in-IP, Flannel backends, GKE alias IPs, Azure CNI options, Cilium networking concepts. Overhead derivations are on the MTU page.
Overlay. Each node has a pod subnet. A packet for a pod on another node is wrapped, sent node to node as ordinary traffic, then unwrapped. It works on any network, at the cost of the MTU and a little CPU. Forgetting to lower the pod MTU produces the classic "small requests work, large ones hang" symptom (debugging by layer).
VPC-native. The pod IP is a real address in your cloud network, so the cloud's routing, security groups and flow logs see pods directly and no encapsulation is needed. The price is address space: see pod IP exhaustion.
Pod egress and source NAT
A pod IP from an overlay or a private pod CIDR is not routable outside the cluster, so traffic leaving it is source NATed to the node's IP (the plugin or an agent such as ip-masq-agent installs a masquerade rule for destinations outside the cluster ranges). The AWS VPC CNI does the same for traffic leaving the VPC unless external SNAT is configured (AWS VPC CNI settings). From there it follows the normal cloud path: private subnet, then a NAT gateway. Many pods behind one node IP means many connections sharing one source IP, so port exhaustion shows up at the node and the NAT gateway.
Replacing kube-proxy
kube-proxy has more than one Linux backend: iptables (the default), IPVS and nftables (kube-proxy modes). iptables evaluates rules as a list, so cost and update time grow with Services and endpoints. IPVS keeps a kernel hash table and offers more balancing algorithms than random (round robin, least connection and others). eBPF dataplanes such as Cilium can replace kube-proxy entirely, doing the Service translation in eBPF programs at the socket or interface (Cilium kube-proxy replacement). All of them still do the same job: map a virtual IP to a pod IP.
Source IP and externalTrafficPolicy
A NodePort or LoadBalancer Service with the default externalTrafficPolicy: Cluster may forward to a pod on a different node, and to make the reply return the same way it SNATs the client IP to the node. The pod therefore sees a node IP, not the client. Setting externalTrafficPolicy: Local sends only to pods on the receiving node and preserves the client IP, at the cost of uneven spread and nodes with no local pod failing the load balancer's health check (Using source IP). Compare keeping the client IP on a normal balancer.
Ingress and Gateway API
A Service of type LoadBalancer is one cloud balancer per Service. Ingress shares one entry point: a controller (nginx, an AWS ALB controller, Envoy-based ones) watches Ingress objects and programs a proxy to route by host and path. The Kubernetes project says Ingress is frozen and new work goes to Gateway API, which split roles into GatewayClass, Gateway and HTTPRoute and reached GA in October 2023 (Ingress docs, Gateway API GA). The design of those proxies is in proxies and gateways.
CoreDNS and the ndots:5 trap
Pods query CoreDNS (reached through the kube-dns Service). The kubelet writes the pod's /etc/resolv.conf with a search list and options ndots:5 (DNS for Services and Pods). The rule: a name with fewer than ndots dots is tried against each search domain before it is tried as written.
api.example.com has two dots, so with a pod in namespace default the resolver asks, in order:
Three wasted lookups per new connection is load on CoreDNS and added latency, and a lost UDP query is retried after the resolver's default 5 second timeout. Fixes: write the name with a trailing dot (api.example.com.), lower ndots for the pod through dnsConfig (for example 2), or run NodeLocal DNSCache so lookups hit a per-node cache first (NodeLocal DNSCache, customizing pod DNS). Resolver mechanics are owned by the DNS page; the debugging view is in debugging by layer.
Conntrack and the 5 second DNS timeout
DNS over UDP through a Service goes through DNAT, so it depends on conntrack. A resolver that sends its A and AAAA queries in parallel from the same socket can create two conntrack entries for the same tuple at the same moment; one insert loses a race and that packet is dropped, which the application sees as a 5 second stall. This was analysed in an influential write-up on racy conntrack and DNS lookup timeouts. NodeLocal DNSCache sidesteps it by letting pods talk to a cache on the same node, which the NodeLocal docs describe as avoiding DNAT rules and connection tracking for those lookups; its upstream connections can also be upgraded to TCP. A full conntrack table breaks Service traffic more broadly; see conntrack and what happens when it fills.
Connection-level balancing and gRPC
kube-proxy chooses a backend when a connection is created, not per request. HTTP/2 and gRPC multiplex many requests on one long-lived connection, so a client connects once and sends everything to one pod. Fixes: a headless Service with client-side balancing, an L7 proxy or mesh that balances per request, or limiting connection age (gRPC load balancing on Kubernetes). The same effect is described generally in load balancing.
NetworkPolicy
A NetworkPolicy selects pods and lists the ingress and egress traffic allowed to them. Pods that no policy selects are not isolated: all traffic is allowed. Once a policy selects a pod for a direction, only what some policy allows gets through, and policies are additive allow rules (NetworkPolicy). Enforcement is done by the CNI plugin; the API server accepts a policy that nothing enforces. The usual start is default deny, then allow:
Denying egress also blocks DNS, so add a rule allowing port 53 (UDP and TCP) to CoreDNS in kube-system. NetworkPolicy works at L3 and L4 by pod label; it is a different tool from cloud security groups.
Pod IP exhaustion on VPC-native clusters
With the AWS VPC CNI each pod takes a secondary IP on a node's network interfaces, so the node's pod limit is ENIs x (IPs per ENI - 1) + 2 (EKS instance type choice). For an instance with 3 interfaces and 10 IPs each that is 3 x 9 + 2 = 29 pods. Beyond that limit pods stay Pending, and a small subnet runs out of addresses before nodes run out of CPU. The levers are prefix delegation (a /28, 16 addresses, per slot), custom networking with a secondary CIDR for pods, and bigger subnets (EKS IP optimization, prefix delegation). Planning room for this is part of the cloud network model. GKE VPC-native clusters give each node a pod range from a secondary range of the subnet, by default a /24 per node with up to 110 pods (GKE alias IPs).
Mesh sidecars
A service mesh adds a proxy next to each pod that intercepts its traffic with iptables rules and adds retries, mTLS and metrics. It sits on top of the CNI and Service layers described here. The mechanism and its costs are owned by proxies and gateways.
Name map
| Concept | AWS (EKS) | GCP (GKE) | Azure (AKS) | Gotcha |
|---|---|---|---|---|
| Pod IP source | VPC CNI: secondary IPs on node ENIs from the VPC subnet | VPC-native: alias IP range per node from a subnet secondary range | Azure CNI gives pods VNet IPs; Azure CNI Overlay and kubenet use a separate pod CIDR | VPC-native designs spend your address space; overlays do not |
| Service type LoadBalancer | Network Load Balancer or Classic, via the AWS Load Balancer Controller or in-tree provider | Passthrough Network Load Balancer | Azure Load Balancer | Each Service can create a billable balancer |
| Ingress | ALB through the AWS Load Balancer Controller | External Application Load Balancer through GKE Ingress or Gateway | Application Gateway for Containers or the app routing add-on | Controller choice sets features and cost; check current docs |
| eBPF dataplane | Cilium as an add-on or in EKS Anywhere | Dataplane V2 (Cilium based) | Azure CNI powered by Cilium | Often replaces kube-proxy and enforces NetworkPolicy |
| NetworkPolicy | Supported by the VPC CNI in current versions, or via Calico and Cilium | Supported on clusters with Dataplane V2 or the Calico option | Supported with Azure NPM, Calico or Cilium | A policy no plugin enforces is silently ignored |
| Per-node pod limit | Instance ENI and IP limits unless prefix delegation | Default 110 pods per node, /24 per node | Set by the plugin mode and a max-pods setting | Hits before CPU does on small subnets |
Check each provider's current documentation before relying on a row: EKS networking, GKE networking overview, AKS networking. For the GCP product view of VPCs see the GCP ACE VPC page.
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
| Containers need NAT to share a host address | Kubernetes mandates no NAT between pods; each pod has its own IP | Simple model: a pod can run anything on any port, and pod-to-pod behaves like host-to-host |
| Address things by IP | Address Services by DNS name | Pod IPs change on every restart or reschedule; the Service IP and name stay put |
| kube-proxy is a proxy that forwards your bytes | It only programs kernel rules; the kernel does the forwarding | Packets never pass through a user-space process, so there is no per-packet process cost (the original userspace mode was removed) |
| A load balancer balances by load | A Service picks a backend per connection, randomly in iptables mode | Kernel rules have no view of backend load; per-request or load-aware balancing needs an L7 proxy or mesh |
| No policy means nothing is allowed | No NetworkPolicy means everything is allowed | Kubernetes starts open so clusters work out of the box; isolation is opt-in |
In production
- Cilium is used as a Kubernetes dataplane, and its project keeps a public list of adopters. Google builds GKE Dataplane V2 on Cilium (GKE Dataplane V2).
- AWS documents pod IP pressure as a design constraint for EKS and recommends prefix delegation and secondary CIDRs (EKS IP optimization).
- Kubernetes itself ships NodeLocal DNSCache as the standard answer to DNS latency and conntrack load (NodeLocal DNSCache).
- The Kubernetes blog carries a well-known explanation of why gRPC defeats connection-level balancing (gRPC load balancing on Kubernetes without tears).
Company-specific figures are omitted here on purpose: no primary source was checked for them.
History and evolution
- 2014 Google open-sources Kubernetes, descended from its internal Borg system, with the rule that every pod has its own IP (Borg, the predecessor to Kubernetes).
- 2015 CoreOS proposes the CNI spec, giving container runtimes one plugin interface so networking can vary without changing Kubernetes.
- 2016 kube-proxy moves from proxying in user space to programming iptables, since a user-space hop per connection did not scale.
- 2018 IPVS mode reaches GA in Kubernetes 1.11, because long iptables chains get slow with thousands of Services (IPVS deep dive). CoreDNS becomes the cluster DNS.
- 2021 EndpointSlices go GA so one pod change no longer re-sends a whole large Endpoints object.
- 2023 Gateway API reaches GA as v1.0, and the Ingress API is frozen (announcement).
- Now Choices are iptables, IPVS, nftables or an eBPF replacement for kube-proxy; managed clouds mostly default to their own VPC-native CNI.
- Next eBPF dataplanes, Gateway API as the standard north-south API, and multi-cluster networking.
Hands-on
List a Service and its endpoints. Output here is illustrative; your addresses will differ.
Check what a pod's resolver does. This is where ndots:5 shows (the search list begins with the pod's namespace). Illustrative:
Resolve a Service name, then compare an external name with and without the trailing dot:
On a node (Linux, as root), find the rules for a Service. IPVS clusters use ipvsadm -Ln instead:
For a throwaway debugging pod with the usual tools, run kubectl run netshoot --rm -it --image=nicolaka/netshoot -- bash, then use dig, curl -v and ss from inside the pod's network namespace. Clusters on macOS laptops run inside a Linux VM, so node-level commands need the VM's shell. The wider method is in debugging by layer.
Gotchas
Interview questions
Show answer
A Service is a stable virtual IP and DNS name in front of a changing set of pods chosen by label. Pod IPs change every time a pod is recreated, so clients that held them would break. The Service gives a fixed target and the cluster keeps the backends up to date.
Follow-ups:
- What types are there? ClusterIP (internal), NodePort, LoadBalancer (cloud balancer), headless (DNS returns pod IPs) and ExternalName (a CNAME).
- What is the NodePort range? The default is 30000-32767.
Show answer
The client resolves the Service name to the ClusterIP and sends to it. On the node, kube-proxy's rules match that IP and port and DNAT the destination to one endpoint pod IP, chosen per connection. Conntrack records the translation, so later packets follow the same path and the reply is rewritten back to the ClusterIP before the client sees it.
Follow-ups:
- iptables versus IPVS? iptables walks a rule list whose size grows with endpoints; IPVS uses a kernel hash table and more algorithms.
- Does kube-proxy touch the packets? No. It only writes the rules; the kernel forwards.
Show answer
The CNI plugin makes that work. An overlay wraps pod packets in an outer packet between nodes. A routed plugin advertises pod subnets so nodes route to each other. A VPC-native plugin gives pods real cloud addresses and the cloud network routes them. In all three the destination pod sees the real source pod IP, with no NAT.
Follow-ups:
- What does an overlay cost? Header overhead: VXLAN takes 50 bytes, so a 1500 underlay leaves 1450 for pods.
- What breaks if you forget it? Large packets are dropped or fragmented while small ones work.
Show answer
Suspect ndots:5: an external name with fewer than five dots is tried against each cluster search domain first, so one lookup becomes several, each asked for A and AAAA. Also suspect the UDP conntrack race that causes 5 second stalls, and CoreDNS being overloaded or uncached.
Follow-ups:
- Fixes? Trailing dot on the name, lower
ndotsviadnsConfig, NodeLocal DNSCache, more CoreDNS replicas. - How do you confirm? Read the pod's
resolv.confand capture port 53 traffic from the pod.
Show answer
NodePort opens one port on every node and is mostly a building block. A LoadBalancer Service asks the cloud for one external balancer per Service, simple but billed each. Ingress (or Gateway API) shares one entry proxy that routes by host and path to many Services, which is cheaper and gives L7 features.
Follow-ups:
- How do you preserve the client IP?
externalTrafficPolicy: Local, or proxy protocol or an L7 header, accepting uneven spread with Local. - Which API for new work? Gateway API; Ingress is frozen.
Show answer
Services balance per connection, and gRPC runs many requests over one long-lived HTTP/2 connection. The client opens it once and kube-proxy picked one backend for it. Fix it with an L7 proxy or mesh that balances per request, or a headless Service so the client resolves all pod IPs and balances itself.
Follow-ups:
- Why is multiplexing the cause? HTTP/2 carries many streams on one TCP connection, so connection-level choice means request-level imbalance.
- A cheaper mitigation? Set a maximum connection age so clients reconnect and re-spread.
Show answer
Apply a default-deny NetworkPolicy for ingress and egress with an empty pod selector, then add allow policies by pod and namespace label for the flows you need. It only works if the CNI enforces NetworkPolicy, so verify that first.
Follow-ups:
- What do people forget? Egress deny blocks DNS, so allow port 53 to CoreDNS.
- Is it a firewall for the node? No. It is L3 and L4 by pod selector, inside the cluster.
Show answer
With the VPC CNI each pod needs a secondary IP on a node's ENIs, so a node holds at most ENIs x (IPs per ENI - 1) + 2 pods, and the subnet itself can run out of addresses. Fix with prefix delegation (a /28 per slot), custom networking with a secondary CIDR for pods, larger subnets, or fewer larger nodes.
Follow-ups:
- What is the worked limit? 3 interfaces with 10 IPs each gives
3 x 9 + 2 = 29pods. - What is the trade-off of an overlay instead? No VPC address use, but encapsulation cost and the cloud no longer sees pod IPs.
Fun facts
- In iptables mode the ClusterIP exists on no interface, only in rules. In IPVS mode kube-proxy binds Service IPs to a dummy interface named
kube-ipvs0, so those addresses can answer locally (IPVS deep dive). - The default Service CIDR in kubeadm is
10.96.0.0/12, and the built-inkubernetesService takes its first address,10.96.0.1(kubeadm reference). - A headless Service has no virtual IP at all, so kube-proxy writes no rules for it.
Cheat sheet
- 30000-32767 - default NodePort range
- ndots:5 - default in a pod's resolv.conf, so names with fewer than 5 dots try search domains first
- 50 bytes - VXLAN overhead, leaving 1450 inside a 1500 underlay
- 100 - default maximum endpoints per EndpointSlice
- ENIs x (IPs per ENI - 1) + 2 - AWS VPC CNI pods per node