Routing Overview - Control Plane, Routing Tables, Convergence
- A router does two jobs. The data plane forwards each packet by looking up its destination in a table. The control plane works out what goes in that table. Routing is the second job, forwarding is the first.
- Routes are stored twice: the RIB holds every candidate from every source, the FIB holds only the winner per prefix in a form hardware can search fast.
- A router never knows the whole path. A table row names only the next hop and the interface to reach it.
- When two sources offer the same prefix, administrative distance picks the source; when one source offers two paths, the metric picks the path. Which vendor numbers apply is vendor-specific.
- Convergence is the time until every router agrees again after a change. During it, packets can blackhole or loop. Detection (BFD, hello timers) usually dominates that time.
Mental model
A GPS app and a driver. The app works out routes using the whole map, takes its time, and re-plans when a road closes. The driver does no planning: at each junction they read the sign that says "toward the airport, turn left" and turn. The control plane is the app, the data plane is the driver at the junction, and the sign is a table row. Nobody in the car, and no junction, holds the full route to the destination. Each junction only knows the next turn.
Where it breaks: a GPS app plans for one car. A router fleet plans for everyone at once, and every router computes (or learns) its own signs, so for a while after a road closes some junctions still point at the old road while others already point at the new one.
It is a build step versus a request path. The control plane is a slow compile that reads configuration and network state and produces a lookup table. The data plane is the runtime that serves every request by reading that compiled table and never recomputing it. The RIB is the source with all candidates, and the FIB is the optimised artifact the runtime loads. A management plane sits beside both, like the deploy tooling: CLI, SSH, SNMP, NETCONF, the ways operators configure and observe the box.
Where it breaks: a compiled artifact is rebuilt by one process. A network's tables are rebuilt by thousands of routers, each acting on a slightly different, slightly stale view, so the "build" is never atomic and mixed states are visible to live traffic.
How it works
RFC 1812 (requirements for IPv4 routers) is where the vocabulary comes from: a router forwards packets using a forwarding table and gets that table from routing protocols or configuration. The split into planes is what lets a router forward at line rate while a slow process worries about topology.
What one routing table row holds
Every row answers one question: for destinations inside this prefix, where does the packet go next? The columns below are the common ones. Names vary by platform, and the FIB keeps only what forwarding needs (prefix, next hop, interface).
| Column | Meaning | Example |
|---|---|---|
| Prefix | The destination block this row covers, in CIDR notation (RFC 4632) | 10.0.0.0/8 |
| Next hop | The adjacent router to hand the packet to; empty for a directly connected network | 10.1.1.2 |
| Interface | The port the next hop is reached through | eth1 |
| Protocol (source) | Which route source installed it: connected, static, or a routing protocol | ospf |
| Metric | That source's own cost for this path; only comparable within one source | 20 |
| Administrative distance | How much the router trusts the source; lower wins between sources | 110 |
The table stores a next hop, not a path. Router A knows "toward 10.0.0.0/8, send to B". It does not know that B will hand the packet to C and then D. That is why hop-by-hop forwarding scales, and also why a disagreement between two routers can create a loop that no single router can see.
Where the longest-prefix lookup fits
For a given destination address the router first finds the most specific matching prefix. That lookup, and how it is made fast, is owned by forwarding and longest-prefix match. The host-side version of the decision (local or via the gateway) is how a host decides where to send a packet. This page covers what happens one step earlier: how the rows got there.
Where routes come from
Four sources fill a routing table:
- Connected: the router assigns an address to an interface, and the attached subnet becomes a route automatically.
- Static: an operator types a route. It never changes on its own and never notices a dead next hop unless it is tied to a health check.
- Dynamic: a routing protocol learns routes from neighbours and recomputes them when the topology changes.
- Default: the catch-all
0.0.0.0/0row. It can come from any of the sources above. It means "send anything I have no better row for to this next hop", nothing more.
Static versus dynamic is a tradeoff between control and adaptation. A static route costs no protocol traffic and cannot be poisoned by a neighbour, so it fits stub networks with one exit, and it is the normal model in cloud route tables (VPC route tables). A dynamic protocol reacts to failures without a human, which is what makes it worth its complexity in networks with several paths.
Metric, cost and choosing between sources
A metric is a number a protocol uses to rank its own paths. Its meaning is private to the protocol: OSPF derives cost from link bandwidth, RIP counts hops (RFC 1058, maximum 15, 16 means unreachable), and BGP does not use one metric at all but a policy-driven comparison list (RFC 4271, see BGP). A hop count of 3 from RIP and a cost of 20 from OSPF cannot be compared, so when both offer the same prefix the router uses administrative distance first. The values below are Cisco IOS defaults, from Cisco's What Is Administrative Distance?. Other vendors use different numbers (and call it route preference), so treat the numbers as vendor-specific and read your platform's documentation.
| Route source | Cisco default AD |
|---|---|
| Connected interface | 0 |
| Static route | 1 |
| eBGP | 20 |
| OSPF | 110 |
| RIP | 120 |
| iBGP | 200 |
The fixed order of decisions (longest prefix, then administrative distance, then metric) and a worked example of a stale static route shadowing an OSPF route are in route sources and administrative distance. The one trick worth knowing here is the floating static route: a static route given a higher distance than the dynamic protocol (for example 200) sits unused in the RIB until the dynamic route disappears, and then becomes the backup.
RIB and FIB
The router keeps every candidate route in the RIB, runs the selection above per prefix, and installs only the winner (or the equal-cost set) into the FIB. They differ because they serve different readers: the RIB is read by protocols and operators and needs every candidate and its attributes, and the FIB is read by forwarding hardware and needs one fast-to-search answer. A route can be in the RIB and not in the FIB (it lost the selection, or the hardware table is full). The recursive-lookup detail is covered in RIB, FIB and recursive lookup.
IGP versus EGP
The internet is too big and too varied to run one routing protocol. Routing is split by who owns the network:
| IGP (interior) | EGP role (exterior) | |
|---|---|---|
| Runs | Inside one organisation's network (one autonomous system) | Between autonomous systems |
| Goal | Best path by a technical metric, fast convergence | Reachability under business policy |
| Protocols | OSPF, IS-IS, RIP | BGP (the original EGP is long retired) |
| Trust | Everyone is one operator | Neighbours are other companies |
An autonomous system (AS, see Autonomous systems) is a set of networks under one administrative policy. The business side of why networks connect to each other (transit, peering) is on edge and core; this module only needs the routing consequence: inside an AS an IGP carries topology, between ASes BGP carries policy-filtered reachability.
Deep dive
Convergence: what happens after a failure
Convergence is the time from a topology change until every router's FIB is consistent again. The sequence has four stages, shown below.
Detection comes first because nothing else can start until a router knows. On a clean physical link loss, the interface goes down at once. On a failure the interface does not report (a switch in the middle dies, a cable still shows carrier), the router must notice missing messages. OSPF declares a neighbour dead after the dead interval, which is 40 s with the common default hello interval of 10 s (the values suggested in RFC 2328, appendix C.3; they are configurable). BGP uses a hold timer: RFC 4271 suggests 90 s with keepalives at a third of that, and vendors choose their own defaults (Cisco IOS commonly uses 60 s keepalive and 180 s hold; this is vendor-specific). BFD (RFC 5880) is a lightweight liveness protocol that any routing protocol can subscribe to. Its detection time is the transmit interval times a detect multiplier, both negotiated, so for example 50 ms times 3 is 150 ms (illustrative values, set per link).
Propagation and recomputation depend on the protocol. A link-state protocol floods a new advertisement to every router, and each reruns its shortest-path calculation (link-state routing). BGP is slower and less predictable: Labovitz and colleagues injected faults into the real internet and measured that inter-domain routers may take tens of minutes to reach a consistent view after a fault, because of oscillations in the path selection process (Delayed Internet Routing Convergence, SIGCOMM 2000). BGP has improved since, but it is why a design that needs fast failover should not wait for it, and why anycast steering cares about BGP convergence.
Installing the result into the FIB is the last step. Until each router finishes it, that router forwards using stale state.
Blackholes and micro-loops
Routers finish at different times, so for a short while the network is inconsistent. Two failures follow:
- Blackhole: a router still sends traffic toward a next hop that can no longer deliver it, so packets vanish until the router updates. A static route to a dead gateway is a permanent blackhole.
- Micro-loop: router A has already switched to a new path through B, but B has not yet updated and still sends the traffic back to A. Packets bounce until one of them updates or the TTL runs out. RFC 5715 is the framework document on this problem.
Mitigations either shrink the window or remove the need to wait. Faster detection (BFD) shrinks the first stage. Loop-free alternates (RFC 5286) precompute a backup next hop that is safe to use the instant the primary fails, so the router can repair locally without waiting for the network. TI-LFA, which uses segment routing to extend that idea to topologies LFA cannot cover, is an IETF work item built on the same principle.
ECMP
When the FIB has several equal-cost next hops for one prefix, the router can use all of them. Packets of one flow must stay on one path or TCP sees reordering, so routers hash fields of the flow (typically source and destination address, protocol and ports, the 5-tuple) and map the hash to a next hop. Every packet of a flow lands on the same path. The mechanics, the elephant-flow imbalance and the Linux hash policy are in ECMP: equal-cost multipath. One further hazard belongs here: if consecutive routers use the same hash function and seed, they pick the same bucket and traffic polarises onto a subset of links. Varying the hash seed per tier is the usual fix.
Redistribution
A router can inject routes learned from one protocol into another (for example OSPF routes into BGP). Because metrics are private to each protocol, the receiving protocol assigns the redistributed routes a fresh metric, and a careless two-way redistribution can feed a route back to where it came from and create a loop. This is a common source of routing outages, which is why redistribution is usually filtered by prefix.
Why one flat IGP does not scale to the internet
Three limits stop it. State: every router would hold every link in the world, and the global table is on the order of a million IPv4 prefixes (live figures at the CIDR Report and bgp.potaroo.net). Churn: a flap anywhere would trigger recomputation everywhere. Policy and trust: competing companies will not give each other their topology or accept a neighbour's shortest-path choice over their own business rules. The answer is hierarchy: an IGP inside each AS to keep convergence fast and state small, and BGP between ASes to carry reachability under policy. CIDR aggregation (RFC 4632) shrinks the table by advertising one covering prefix in place of many.
SDN: a centralised control plane
In software-defined networking, the control plane is lifted out of each box into a logically central controller that computes routes with a global view and programs the forwarding tables of the devices, which keep only the data plane. The attraction is that one program sees the whole topology, so there is no distributed convergence race. The cost is that the controller is a new critical dependency. The layering is described in RFC 7426.
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
| Routers compute shortest paths | At the internet edge, BGP chooses by policy (local preference before AS-path length); shortest path applies only inside an IGP | Networks are run by competing businesses, and cost and relationships matter more than hop count |
| Convergence is a quick settling-down | Detection alone can be 40 s with default timers and BGP can take minutes, so operators add BFD and precomputed backups | Protocol timers were set conservatively to avoid false alarms on slow links |
| The routing table is the forwarding table | The RIB (all candidates) and FIB (winners) are separate, and the FIB lives in size-limited hardware memory | Forwarding needs a fixed-time lookup, so hardware capacity bounds how many routes a device can forward on |
| Static routes are for toys | Cloud route tables and data centre default routes are mostly static plus ECMP | Where topology is known and controlled, a static route is simpler and cannot be corrupted by a neighbour |
In production
- Google publishes how it moved parts of the control plane into software: Jupiter Rising covers centralised control in its data centre network, and Espresso describes an SDN peering edge that decides per-prefix egress using global information.
- Large data centres often use BGP alone as the routing protocol with ECMP across many equal paths. RFC 7938, written by engineers at Facebook and Microsoft, explains the design and why they picked BGP over an IGP at that scale.
- AWS VPC route tables are static: you add routes and the table applies longest-prefix match, with a documented priority order for equal prefixes (route table documentation; the model is on VPC routing).
- Anycast operators announce the same prefix from many sites and let BGP pick the closest one, so their failover speed is bound by BGP convergence (global traffic steering).
History and evolution
- 1969 The ARPANET starts with distance-vector routing: each node tells neighbours its distance to every destination and applies Bellman-Ford. It is simple and needs no global view.
- 1979 Slow convergence and routing loops in the distance-vector scheme lead the ARPANET to a new algorithm: link-state routing, where each node floods link status and computes shortest paths itself (McQuillan, Richer and Rosen, 1980).
- 1984 RFC 904 specifies EGP, the first protocol between autonomous systems. It assumes a tree-shaped internet, and that assumption breaks as networks interconnect freely.
- 1988 RFC 1058 documents RIP, distance-vector with a hop-count metric capped at 15, which stops scaling past small networks.
- 1989 BGP version 1 (RFC 1105) replaces EGP for inter-AS routing, and the link-state IGP OSPF begins its standardisation. OSPFv2 is RFC 2328.
- 1993 Address exhaustion and a growing table force CIDR (RFC 1519, now RFC 4632), so routes can be aggregated at any prefix length.
- Late 1990s to 2000s MPLS traffic engineering and faster-failure tooling (BFD, RFC 5880) arrive to fix the slow detection and convergence that plain protocol timers leave.
- 2008 onward OpenFlow and SDN move the control plane into software controllers, and hyperscalers adopt them for data centres and peering edges.
- Now BGP plus an IGP plus ECMP is the standard shape of production networks, and the IPv4 table is on the order of a million prefixes (live: bgp.potaroo.net).
- Next Segment routing, including over IPv6 (RFC 8986), moves path choice into the packet, and intent-based tooling aims to generate configuration from declared goals.
Hands-on
macOS first, because this is what the author's machine prints. netstat -rn shows the table and route -n get shows which row a destination uses. The output is abbreviated (neighbour entries removed) from macOS 27 on a home network.
Read the Gateway column: an address is a next hop, and link#15 means directly connected (no gateway, resolve the destination with ARP). Flags U (up), G (gateway) and S (static) are the ones to know. The single default row is why this host reaches everything beyond its LAN through 192.168.0.1.
The destination printed is default: 8.8.8.8 matched no more specific row, so the default route was chosen. This is the longest-prefix decision made visible.
On Linux, the equivalents are ip route and ip route get. The output below is illustrative, not captured here; the proto field names the route source:
proto kernel is a connected route, proto dhcp came from a DHCP lease, and proto ospf or proto bgp means a routing daemon installed it. IPv6 has its own table (ip -6 route, or netstat -rn -f inet6 on macOS). On Windows, route print shows the table. See ss, nc, lsof and ip for the full ip tool, and traceroute to see the hops a route produces.
Gotchas
Interview questions
Show answer
The control plane decides where traffic should go: it runs routing protocols and builds the routing table. The data plane moves each packet by looking its destination up in the forwarding table, usually in hardware. Routing is the first, forwarding is the second. Control plane work takes seconds, the per-packet data plane work takes nanoseconds.
Follow-ups:
- What is the management plane? How operators configure and monitor the device: CLI, SSH, SNMP, NETCONF.
- What changes with SDN? The control plane moves to a central controller and devices keep only the data plane.
Show answer
A destination prefix, the next hop (or none if directly connected), the outgoing interface, the source protocol, and a metric. Most platforms also store an administrative distance. The router matches the packet's destination against the prefixes and sends the packet to that row's next hop.
Follow-ups:
- What is the default route? The
0.0.0.0/0row, matched when nothing more specific matches.
Show answer
First the longest matching prefix wins. If two sources offer the same prefix, the lower administrative distance wins. If one source offers two paths for the same prefix, the lower metric wins. Equal metrics can be installed together as an ECMP set.
Follow-ups:
- What if the paths are equal cost? The router installs both and spreads flows with a hash.
- What is a floating static route? A static route with a higher distance than the dynamic one, so it is a backup.
Show answer
Convergence is the time for every router to reach a consistent view after a change. While it lasts, some routers use old routes and some new ones, so packets can blackhole at a dead next hop or loop between two routers that disagree (a micro-loop). The time is detection plus propagation plus recomputation plus FIB install, and detection often dominates.
Follow-ups:
- How do you speed it up? BFD for fast detection, loop-free alternates for precomputed backups, and tuned SPF timers.
- Why is BGP slow? Policy and path exploration; Labovitz et al. measured tens of minutes in 2000.
Show answer
Static fits small or stub networks with a single exit and environments where topology is controlled, such as cloud route tables and data centre defaults: no protocol overhead and nothing a neighbour can corrupt. Dynamic fits networks with redundant paths, where the network must reroute around failures without a human.
Follow-ups:
- How do you get the best of both? A floating static route as a backup behind a dynamic route.
Show answer
State, churn and policy. Every router would hold every link, and any flap anywhere would trigger recomputation everywhere. Independent companies also will not share topology or accept a single metric. So the internet uses hierarchy: an IGP inside each autonomous system and BGP between them, with CIDR aggregation to keep the table small.
Follow-ups:
- How big is the BGP table? On the order of a million IPv4 prefixes; read the live figure at bgp.potaroo.net.
Show answer
Equal-cost multipath installs several next hops for one prefix and spreads traffic across them. To avoid reordering inside a connection, the router hashes the flow identifier (the 5-tuple) and maps the hash to a next hop, so one flow always takes one path. The cost is imbalance: the hash balances flows and not bytes.
Follow-ups:
- What is polarisation? Successive routers using the same hash pick the same bucket and overload some links.
- How do you fix it? Use a different hash seed per tier.
Fun facts
- The ARPANET started with distance-vector routing in 1969 and changed to link-state in 1979 after convergence and looping problems (McQuillan et al.).
- RIP treats 16 hops as infinity, so a network more than 15 routers across is unreachable to it (RFC 1058).
- The size of the IPv4 global table is public and changes daily, with live graphs at bgp.potaroo.net.
Cheat sheet
| Step | Rule | Decides between |
|---|---|---|
| 1 | Longest matching prefix | Routes for different prefixes |
| 2 | Lowest administrative distance | Same prefix, different sources |
| 3 | Lowest metric | Same prefix, same source |
| 4 | ECMP hash per flow | Still tied: several equal next hops |