Debugging Cloud Networks - Flow Logs, Reachability, Mirroring
- A timeout means something silently dropped the packets (route, security group, NACL). Refused means the packet reached a host and nothing was listening. Start every cloud debug by deciding which one you have.
- You cannot run
tcpdumpon the router or the hypervisor, so the cloud gives you three substitutes: flow logs (per-interface records of what was allowed or rejected), path analyzers (read the configuration and name the blocking component, no packets sent), and traffic mirroring (a real packet copy of one interface). - In a flow log,
REJECTmeans a security group or NACL denied it, but not which one. A loneREJECTpoints at the security group or the inbound NACL. AnACCEPTfor the request followed by aREJECTfor the reply points at a stateless NACL. - Flow logs are aggregated (1 or 10 minutes on AWS) and exclude some traffic, so absence of a record is weak evidence.
- Reachability Analyzer and Connectivity Tests prove a path from configuration. They cannot see a closed port, an OS firewall or a crashed app.
Mental model
A parcel that never arrives. You cannot stand in the depot yourself, but the tracking page shows each depot's scan: "left sorting centre A", "refused at depot C: no such address". A missing scan means the parcel was lost before that depot, a "refused" scan names where it was turned back, and "delivered, nobody home" is a different problem from "never arrived".
Where it breaks: a tracking page is near real time and covers every parcel. Flow logs arrive minutes late, summarise packets into per-flow lines, and skip some kinds of traffic entirely, so a missing scan is not proof the parcel never moved.
This is tcpdump at every hop, except you may not log into any hop. The hypervisor, virtual router and firewall are the provider's machines, so instead you read the logs they emit (flow logs), ask the provider to evaluate your configuration offline (Reachability Analyzer, Connectivity Tests, Network Watcher) or ask for a copy of one interface's traffic (mirroring). The generic method is the layer ladder in debugging by layer; this page is its cloud specialisation.
Where it breaks: a path analyzer is a static check of rules and routes, like a type checker. It says the program is well formed, not that it behaves at run time. A host with nothing listening on the port passes the analysis and still refuses the connection.
How it works
The cloud hides the middle of the path, so the work is choosing the right instrument for the symptom. The first split is the one from the layer ladder: does the connect attempt time out, get refused, or succeed with a wrong or slow answer? Tools for the three outcomes differ.
Worked example: web cannot reach db on 5432
The web instance 10.0.1.5 in a public subnet cannot reach a Postgres instance 10.0.21.9 in a private subnet. Work it in four steps.
Step 1: classify the failure. From the web instance:
It hangs until the 5 second bound and then times out. It did not fail at once, so nothing sent a reset: a packet was dropped, or the route does not exist. A fast Connection refused would instead mean the path is fine and the database is not listening (error messages to cause).
Step 2: read the flow log. With flow logs enabled on the database's interface, a rejected attempt shows up as a line like this (an illustrative record, not captured output):
| Field | Meaning | Example |
|---|---|---|
| version | Flow log record format version | 2 |
| account-id | AWS account that owns the interface | 123456789012 |
| interface-id | The network interface (ENI) that recorded the flow | eni-0a1b2c3d |
| srcaddr, dstaddr | Source and destination IP of the traffic | 10.0.1.5, 10.0.21.9 |
| srcport, dstport | Source and destination port. The client side is an ephemeral port | 49152, 5432 |
| protocol | IANA protocol number: 6 is TCP, 17 is UDP, 1 is ICMP | 6 |
| packets, bytes | Count in the aggregation window, not per packet | 3, 180 |
| start, end | Window start and end as Unix seconds. Here the window is 60 s | 1700000000, 1700000060 |
| action | ACCEPT if a security group and NACL permitted it, REJECT if not | REJECT |
| log-status | OK, NODATA (no traffic in the window) or SKIPDATA (records were skipped) | OK |
Read it: 10.0.1.5 from ephemeral port 49152 to 10.0.21.9 port 5432 over TCP, a 60-second window, 3 packets of 180 bytes in total, rejected. Three 60-byte packets is what a client that never gets an answer sends: a SYN and two retransmits. A record exists at all, so the packet reached the database's interface and was denied there: the route is fine, a filter is not. If no record had appeared for either interface, suspect the route (a blackhole or a missing entry) or a log delay, not a firewall.
Step 3: decide which filter. A security group is stateful and a NACL is stateless, so the shape of the log tells them apart:
- A single
REJECTfor the request, with noACCEPTbefore it, means the packet never passed the front door: the database's security group (does it allow the web tier on 5432?) or the inbound NACL on the database subnet. - An
ACCEPTfor the request and then aREJECTfor the reply (source port 5432, destination port 49152) means the request got in and the answer was stopped at a stateless NACL that lacks the ephemeral-port rule. A security group never rejects the reply to a connection it allowed. Security groups vs NACLs works that arithmetic out.
Step 4: prove the fix without sending packets. Ask the analyzer to evaluate the path from the web ENI to the database ENI on port 5432. It either reports the path as reachable with the hop list, or not reachable with an explanation naming the component, such as a security group that has no matching inbound rule (Reachability Analyzer docs). Fix the named rule, rerun the analysis, then confirm with the same nc command, because only a real connect proves the port is open.
Where the instruments sit on a path
Deep dive
What flow logs do not record
Flow logs are a record of accepted and rejected flows at an interface, not a packet capture. AWS documents these exclusions: traffic to the Amazon DNS server (queries to your own DNS server are logged), Windows licence activation traffic, instance metadata traffic to 169.254.169.254, Amazon Time Sync traffic to 169.254.169.123, DHCP traffic, and traffic to the reserved address of the default VPC router (VPC Flow Logs limitations). So "no flow record" never proves "no traffic". For DNS, turn on resolver query logging instead (Route 53 Resolver query logs) and use dig.
Records are also aggregated. The AWS maximum aggregation interval is 10 minutes by default, with a 1-minute option (on Nitro-based instances it is always 1 minute or less, per the flow log limitations), so one line summarises many packets and arrives after the window closes. You can see that a flow was rejected, not the packet order, flags or payload. Records can be published to CloudWatch Logs, S3 or Data Firehose.
REJECT: security group or NACL
The action field is REJECT when traffic was not permitted by the security groups or network ACLs (flow log record examples), and the record does not say which. Use the pairing rule from the worked example: a lone REJECT on the request is the security group or inbound NACL, while an ACCEPT followed by a REJECT on the return is a stateless NACL. The AWS ping example in the same document shows the shape: the inbound echo request is ACCEPT and the echo reply is REJECT when the NACL denies outbound ICMP.
Querying flow logs
At volume you query rather than read. CloudWatch Logs Insights can parse the default fields, and the same logs in S3 can be queried with Athena. Start from the rejects to one destination port and group by source. The query in Hands-on shows the shape.
Path analyzers: what they prove and what they do not
- AWS Reachability Analyzer evaluates a source and destination you name, over a protocol and port, and reports a hop-by-hop path or the component that blocks it. It analyses configuration and does not send packets (docs).
- AWS Network Access Analyzer answers a different question: not "can A reach B" but "what network access exists that should not", by matching your network against scopes you define, for example internet-reachable paths to a database tier (docs).
- GCP Connectivity Tests (in Network Intelligence Center) also analyse configuration, and for some test types add a live data-plane check (docs). Firewall Insights reports which firewall rules are shadowed or unused (docs).
- Azure Network Watcher has narrow single-question tools: IP flow verify (would this 5-tuple be allowed or denied, and by which NSG rule), next hop (which route applies) and connection troubleshoot (overview, IP flow verify).
None of them sees a process that is not listening, an OS firewall such as iptables inside the guest, or a name that resolves to the wrong address. After they say "reachable", test the last mile with ss, nc, lsof and ip and curl and openssl.
Traffic mirroring: real packets
When you need the actual bytes, mirror an interface. AWS Traffic Mirroring copies traffic from a supported instance's ENI, filters it, and sends it VXLAN-encapsulated (UDP port 4789) to a target such as another ENI or a Network Load Balancer in front of a sensor fleet (docs). Open the capture in tcpdump and Wireshark, which decodes VXLAN. GCP Packet Mirroring clones traffic from selected instances, subnets or tags to collectors behind an internal load balancer (docs). Azure Network Watcher offers packet capture on a VM through an extension (overview). Mirrored traffic is extra load on the source and the sensor, so scope the filter narrowly and switch it off after the capture.
GCP and Azure flow logs
| Cloud | Where enabled | Sampling and interval | Source |
|---|---|---|---|
| GCP | Per subnet | Default sample rate 0.5 (50%) for configurations made with the Compute Engine API (100% via the Network Management API), default aggregation interval 5 seconds | VPC Flow Logs |
| Azure (VNet flow logs) | Per virtual network, subnet or NIC, through Network Watcher | See the docs for the current recording interval and retention options | VNet flow logs |
| Azure (NSG flow logs) | Per NSG | Version 2 adds throughput data. Microsoft has announced a move to VNet flow logs, so check the retirement notice before building on it | NSG flow logs |
All three clouds log the flow's 5-tuple, bytes and packets, and the verdict. Field names differ, and GCP adds instance and VPC metadata to each record. Defaults above are as of 2026-10; the linked pages are authoritative.
Cross-cloud name map
| Concept | AWS | GCP | Azure | Gotcha |
|---|---|---|---|---|
| Per-interface flow records | VPC Flow Logs | VPC Flow Logs | VNet flow logs, NSG flow logs | All are aggregated and sampled or windowed, never per packet |
| Prove a path from config | Reachability Analyzer | Connectivity Tests | Connection troubleshoot, IP flow verify, next hop | Static analysis: cannot see a closed port or guest firewall |
| Find unintended access or rule hygiene | Network Access Analyzer | Firewall Insights | NSG diagnostics | Different questions, so the three are not exact equivalents |
| Packet copy | Traffic Mirroring (VXLAN) | Packet Mirroring | Packet capture (VM extension) | AWS and GCP mirror at the platform, Azure captures inside the VM |
| DNS query log | Route 53 Resolver query logs | Cloud DNS logging | Azure DNS analytics | Flow logs skip the VPC resolver on AWS |
| Shell without a public IP | SSM Session Manager | IAP TCP forwarding | Azure Bastion | Needs an agent, an IAM grant or a bastion host in the VNet |
Access without a public IP
A private instance has no address to SSH to. Each cloud offers an identity-checked tunnel instead: SSM Session Manager (an agent on the instance and an IAM permission, no inbound port), GCP IAP TCP forwarding and Azure Bastion. They also give you a vantage point inside the network for the nc, curl and dig checks in this page. If the agent or tunnel itself fails, that is a network problem too: it needs an outbound path to the service, often through a NAT gateway or an interface endpoint.
Symptom to cause
| Symptom | Likely cause | First check | Tool | Owner page |
|---|---|---|---|---|
| Connect times out, same subnet works | Security group or NACL drop | Flow log action on the target ENI | Flow logs, Reachability Analyzer | SG vs NACL |
| Instance has a public IP but is unreachable | No route to the internet gateway, or a missing inbound rule | Route table of the subnet, then the security group | Reachability Analyzer, next hop | VPC routing |
| Connection refused at once | Nothing listening, wrong port or bound to loopback | ss -ltn on the server | ss, nc | error messages |
| Internet works, S3 or a service does not | Endpoint route, endpoint policy or DNS answer | Route table prefix list, then dig the service name | dig, flow logs | private service endpoints |
| Private instance cannot reach the internet | NAT gateway missing, wrong route or exhausted | Default route target and NAT gateway state | Flow logs, NAT metrics | NAT gateways |
| Peered or VPN network unreachable one way | Missing return route or overlapping CIDR | Route tables on both sides | Flow logs on both ENIs | private connectivity |
| Ping fails but the service works | ICMP not allowed | Security group has no ICMP rule | nc to the service port | debugging by layer |
| Small requests work, large ones hang | MTU black hole | Ping with the do-not-fragment flag at growing sizes | ping, tcpdump | debugging by layer |
Textbook vs real world
| The textbook says | Production does | Why |
|---|---|---|
Capture with tcpdump somewhere on the path | Read flow logs, or mirror one interface to a sensor | The path is the provider's virtual network. There is no host in the middle to log in to |
ping the host to see whether it is up | Test the service port with nc or curl | Security groups block ICMP unless you allow it, so a failed ping proves nothing |
traceroute shows each hop | Expect one hop or odd gaps and use route tables plus flow logs | The virtual router is not a normal TTL-decrementing hop (private connectivity) |
| Logs show what happened right now | Treat flow logs as minutes late and aggregated | Records are produced per window (1 or 10 minutes on AWS) |
| A firewall denial is always visible | A drop shows as a REJECT only if logging is on at that interface | Logging must be enabled before the incident, not during it |
In production
- Enable flow logs before you need them. A flow log only records from the moment it is created, so it cannot explain an incident that happened before it existed. Flow logs publish to CloudWatch Logs, S3 or Data Firehose, and the destination has its own charges, so check the current pricing pages (VPC Flow Logs docs) and choose the 1-minute or 10-minute interval deliberately.
- Runbooks lead with the symptom. The pattern is "timeout, then flow log, then analyzer" and "refused, then
sson the host". The symptom table is the template. - Cloud-level failures happen too. On 7 December 2021, AWS reported that an automated activity to scale capacity of one of its services in the main AWS network triggered unexpected behavior from a large number of clients inside the internal network, whose surge of connection activity congested the devices between the two networks in us-east-1, and that real-time monitoring and operational tooling were impaired during the event (AWS post-event summary). The lesson for debugging: keep diagnostic access that does not depend on the thing that is failing.
History and evolution
- 2015 AWS launches VPC Flow Logs, because customers in a virtual network could no longer capture on the wire themselves and needed a record of accepted and rejected traffic.
- 2017 Azure Network Watcher arrives, bringing IP flow verify, next hop and packet capture to a cloud where each of those had been a support ticket.
- 2019 AWS Reachability Analyzer and GCP Connectivity Tests (Network Intelligence Center) appear, moving from "read the logs after the fact" to "analyse the configuration without sending traffic". AWS Traffic Mirroring ships the same year for real packet copies. Treat the exact launch dates as approximate and confirm against each vendor's release notes before quoting them.
- Now Flow logs, path analysis and mirroring exist in all three clouds, though Azure is moving from NSG flow logs to VNet flow logs. Each cloud's current tool list is on the Network Watcher, Network Intelligence Center and VPC pages linked above.
- Next Automated path analysis plus network observability built on eBPF, for example Cilium Hubble for Kubernetes, which sees flows inside the cluster where cloud flow logs stop.
Hands-on
Turn on flow logs for a VPC and send them to CloudWatch Logs (the role and log group names are placeholders):
Find who is being rejected on port 5432 with a CloudWatch Logs Insights query over the default-format fields:
Ask the analyzer instead of guessing, as a path from one ENI to another on a TCP port:
The same questions on GCP and Azure:
Finish with the host-side checks that no analyzer can do. On the server, ss -ltn shows what is listening; from the client, nc -zv -w 5, curl -v and dig test the port, the service and the name:
ss is Linux only; on macOS use lsof -nP -iTCP:5432 -sTCP:LISTEN. nc -z -w behaves the same on both. Confirm the exact flag names for the cloud CLIs against your installed version with --help, as they change between releases.
Gotchas
Interview questions
Show answer
Refused means a machine received the SYN and answered with a reset, so the path works and nothing is listening on that port. Timed out means no answer ever came, so something dropped the packets on the way, such as a firewall, a missing route, or a host that is gone.
Follow-ups:
- Which implies a firewall? A timeout. Most cloud firewalls drop silently rather than reject.
- Can a firewall cause refused? Yes, if it is configured to send a reset, but cloud security groups and NACLs do not.
Show answer
First, classify the failure with nc -zv or curl: timeout or refused. For a timeout, check the route table has a route to an internet gateway, then the security group's inbound rule, then the NACL in both directions, then the OS firewall. For refused, check the service is listening and bound to the right address. Confirm the instance really has a public address.
Follow-ups:
- How do you test each hop without SSH? Use Reachability Analyzer for route and filters, flow logs on the instance ENI for accept or reject, and Session Manager for a shell.
- What about a stopped service? The analyzer will not catch it, so test the port from inside the VPC.
Show answer
Security groups are stateful and NACLs are stateless. A single REJECT on the request means it was denied at the front door, by the security group or inbound NACL. An ACCEPT for the request and a REJECT for the reply means the request was allowed in and a NACL blocked the response. Then confirm with Reachability Analyzer, which names the component.
Follow-ups:
- What is the typical NACL mistake? Missing the ephemeral port range (1024-65535 is the safe union) for return traffic.
- Why can a security group not reject a reply? Because it tracks the connection and lets replies back automatically.
Show answer
For a gateway endpoint, check the subnet's route table has the S3 prefix list route to the endpoint, and that the endpoint policy and any bucket policy allow the call. For an interface endpoint, check the private DNS name resolves to the endpoint's private IP and that the endpoint security group allows 443 from the app. Use dig for the name and flow logs for a reject. See private service endpoints.
Follow-ups:
- What is the prefix list? A managed list of the service's address ranges that the route and security group rules can reference.
- How can the policy block you? An endpoint or bucket policy can deny a request that the network path allows.
Show answer
Resource exhaustion on a stateful device rather than a configuration error, since config errors fail every time. Candidates: NAT gateway port allocation exhausted, connection-tracking limits on the instance, ephemeral port exhaustion on the client, and MTU or fragmentation issues that only hit large transfers.
Follow-ups:
- Which metrics do you graph? NAT gateway
ErrorPortAllocationand active connections, the interface's connection-tracking counters (the ENA driver exposesconntrack_allowance_exceededthroughethtool -S), retransmits, andss -ssocket counts. - How do you separate them? Whether failures cluster on one NAT gateway, one instance, or one destination. See NAT gateways and egress.
Show answer
Compare the failing and healthy zones: if instances in the bad zone cannot reach each other or the internet but their processes are healthy, it is the network or the provider. Check the provider health dashboard, flow logs for rejects or missing traffic, and whether the app has a hidden cross-AZ dependency such as a single-zone NAT gateway or database that makes it fail with the zone.
Follow-ups:
- What cross-AZ dependency is classic? A NAT gateway in one zone used by subnets in all zones.
- How do you keep access during the outage? An access path that does not depend on the failing zone or service, as in the December 2021 summary above.
Show answer
Run Reachability Analyzer from an ENI in one VPC to an ENI in the other on the target port. It evaluates routes, peering or transit gateway attachments, security groups and NACLs, and either returns the path or the blocking component. Because no packet is sent, it is safe in production.
Follow-ups:
- What are its limits? It reads configuration only. It cannot see a closed port, a guest firewall, or DNS answers, so follow it with a real connect test.
- What if the CIDRs overlap? Peering cannot be established between overlapping ranges, so the path fails at routing. See private connectivity.
Fun facts
- Flow log protocol numbers are the IANA protocol numbers: 6 is TCP, 17 is UDP and 1 is ICMP.
- Traffic Mirroring uses VXLAN on UDP port 4789 to carry the copied packets to the sensor.
- The AWS VPC Flow Logs
log-statusfield has three values:OK,NODATAandSKIPDATA. - Queries to the Amazon DNS server are one of the documented flow log exclusions, so DNS needs its own log.
Cheat sheet
- Did the packet get denied, and where? Flow logs on the target interface, read
action. - Does a path exist on paper? Reachability Analyzer, Connectivity Tests, or Azure connection troubleshoot.
- What exactly is on the wire? Traffic Mirroring or Packet Mirroring into tcpdump and Wireshark.
- Is anything listening?
ss -ltnon the host (ss, nc, lsof and ip). - Is the name right? dig and resolver query logs.
GCP product views of the underlying pieces are in GCP ACE: VPC and firewall rules. The generic method is debugging by layer; the cloud model is cloud network model.