Network Troubleshooting: Commands and a Systematic Workflow

Quick Answer

Network troubleshooting is the systematic process of identifying, isolating, and resolving connectivity or performance failures in computer networks. To solve a network issue quickly, follow a top-down or bottom-up layered approach: first verify local interface health, then check IP reachability, test DNS name resolution, trace packet routing paths, validate transport layer port states, and finally inspect application-layer responses. By isolating the exact layer where packets drop or responses fail, you eliminate guesswork and resolve production outages faster.

Define the Symptom

Before executing a single diagnostic command, you must accurately define the symptom. Is the failure intermittent or total? Does it affect a single client container, an entire Kubernetes cluster, or external public users? Gathering exact error messages, HTTP status codes, and exit codes prevents you from chasing ghost problems.

Start by documenting the exact nature of the failure. For instance, an error stating 'Connection refused' means the target host actively rejected the TCP handshake on that port, implying the service is down or firewalled. Conversely, a 'Connection timed out' error implies that packets are dropping somewhere along the path without an ICMP reply, or a stateful firewall is silently dropping incoming SYN packets. Recording whether the issue triggers during DNS lookup, TCP handshake, or TLS negotiation immediately narrows your search space.

Check Local Connectivity

Always begin your active investigation locally. Before checking external gateways or public DNS servers, verify that your local network interface is up, has the correct IP assignment, and can reach its immediate gateway. Misconfigured local loopback interfaces, bad VLAN tags, or downed virtual network adapters frequently masquerade as complex remote outages.

Use the modern ip command-line utility to inspect interface states and routing tables:

ip a
ip route show

Look for the UP flag on your primary interface and confirm that your default gateway is present. Next, test local link reachability using ping against your local gateway:

ping -c 4 192.168.1.1

If local gateway ping fails, the issue is isolated to your immediate local network, local interface configuration, or physical cable/virtual bridge connection.

Test DNS

Once local connectivity is confirmed, check name resolution. Many connection failures that appear to be network drops are actually DNS failures—such as stale cache entries, misconfigured resolv.conf files, or unreachable upstream nameservers.

Distinguish between recursive lookups handled by local resolvers and authoritative lookups handled by domain name servers. Use nslookup or dig to query specific records:

nslookup api.internal.net
dig +trace api.internal.net

If nslookup returns an NXDOMAIN error or times out, verify your /etc/resolv.conf file or test alternative public resolvers like 1.1.1.1 or 8.8.8.8 to determine whether the failure lies within your local resolver infrastructure or the authoritative nameserver itself.

Trace the Route

When local connectivity and DNS resolution succeed, but traffic cannot reach a remote host, you must inspect the intermediate network path. Packets traverse multiple routers, switches, and firewalls before reaching their destination. A bottleneck, misconfigured static route, or MTU black hole can disrupt traffic mid-stream.

Use traceroute (or tracepath) to examine the hop-by-hop path packets take:

traceroute -m 30 -T -p 443 destination.example.com

Note that many modern firewalls drop ICMP echo or UDP traceroute probes for security reasons, resulting in asterisks (*) in the output even when traffic flows successfully. Utilizing TCP-based traceroute flags (-T) often bypasses strict ICMP filtering, giving you accurate visibility into transport-layer routing behavior.

Test the Port

After mapping the route, you must verify transport layer reachability. An IP address may be fully reachable via ping, but the specific port required by your application might be closed, filtered by a security group, or unlistening.

Examine local listening sockets on your server using the socket statistics tool ss:

ss -tulpn | grep LISTEN

To test whether a remote port is open and accepting TCP connections from your client machine, use netcat (nc):

nc -zv target-ip-address 443

A successful connection returns a 'Connection to target-ip-address port 443 [tcp/https] succeeded!' message. If this times out or is refused, inspect intermediary firewalls, cloud security groups, or container port-mapping configurations.

Test the Application

With network routes and transport ports verified, shift focus to the application and session layers. This involves validating HTTP/HTTPS request handling, header responses, and TLS certificate handshakes.

Use curl with verbose logging to inspect the complete exchange:

curl -Iv https://api.example.com/healthz

Inspect the output for TLS handshake success, cipher negotiation, valid certificate chains, and HTTP status codes (such as 200 OK, 502 Bad Gateway, or 504 Gateway Timeout). If curl receives a 502 error, the network path is fully functional, but the upstream application server or reverse proxy is failing internally.

Useful Commands

Having a firm grasp of core diagnostic utilities accelerates your debugging workflow. Below is a categorized quick-reference guide for the essential commands used in network diagnosis.

• ping: Tests ICMP echo reachability and measures round-trip latency. Syntax: ping -c 4 <host>.
• traceroute: Maps packet hops and routing paths. Syntax: traceroute -T <host>.
• nslookup: Queries DNS nameservers for record resolution. Syntax: nslookup <domain>.
• dig: Performs advanced DNS queries with trace support. Syntax: dig +trace <domain>.
• ss: Inspects socket statistics, TCP states, and listening ports. Syntax: ss -tulpn.
• nc (Netcat): Tests TCP/UDP port connectivity and sends raw socket data. Syntax: nc -zv <host> <port>.
• curl: Transfers data with URL syntax, inspecting HTTP headers and TLS handshakes. Syntax: curl -Iv <url>.
• ip: Manages network interfaces, IP addresses, and routing tables. Syntax: ip a.

Troubleshooting Examples

Consider a realistic developer scenario: an internal web application running in a Kubernetes cluster cannot reach an external database service. The error log reports persistent database connection timeouts.

1. Check Local Pod Connectivity: Execute kubectl exec into the application pod and verify network configuration with ip a and ip route.
2. Test DNS Resolution: Run nslookup database.internal.svc.cluster.local inside the container to confirm the internal DNS resolver returns the correct cluster IP.
3. Test the Port: Run nc -zv database.internal.svc.cluster.local 5432 to verify if the PostgreSQL port is reachable. If the connection times out, inspect Kubernetes NetworkPolicies and security groups.
4. Test the Application: Use an interactive database client or specialized tool to verify authentication and TLS parameters once port connectivity is established.

Layered Diagnosis and Best Practices

Systematic network troubleshooting relies on disciplined layer-by-layer verification. Avoid the common mistake of confusing IP addresses with ports, assuming TCP and UDP behave identically, or skipping local checks before debugging complex routing issues.

Never disable firewalls or bypass TLS certificate verification as permanent troubleshooting steps; doing so introduces severe security vulnerabilities. Always verify your changes against authoritative documentation and maintain strict isolation between test and production environments.