Playbook

Is It Down, Or Is It Just Me?

Most outage reports arrive as a single data point: one person, one request, one failure. Turning that into a location takes four or five checks in a specific order, each chosen to eliminate as much as possible. The order matters more than the tools — running them in the wrong sequence means gathering evidence that does not narrow anything.

1. Does the name resolve, and to what?

Start here because everything downstream depends on it and it is the fastest check available. Three outcomes, three different investigations: no resolution means the problem is DNS and nothing else matters yet; resolution to an unexpected address means the problem is that traffic is going somewhere wrong, which is more alarming than it being down; resolution to the expected address eliminates DNS entirely and you can stop thinking about it.

Query a public resolver rather than relying on your own cache, or you risk confirming an answer that no longer exists.

2. Is the host reachable at all?

Ping establishes whether packets are getting there and coming back, and gives you a latency figure to compare against normal. Read the loss carefully — a couple of percent is not negligible, because TCP responds to loss by backing off, so it presents as severe slowness rather than as failure. Remember that a host may be filtering ICMP entirely, in which case silence here proves nothing and you move on rather than concluding it is down.

3. Is the port open?

This is the highest-value check in the sequence because its three outcomes map onto three different teams. Connection refused means something answered and declined — the host is up, the service is not listening, and it is an application problem. Timeout means nothing answered — a firewall or routing issue between you and it. Connected means the network path and the listener are both fine and the fault is above the transport layer.

Most investigations that sprawl across three teams for an hour would have been cut in half by running this in the first two minutes.

4. If it is the path, where?

Only reach for a traceroute once you have established the problem is between the endpoints, and prefer a tool that samples repeatedly rather than taking one measurement per hop. A single traceroute cannot distinguish a router that deprioritises responses about itself from a genuinely broken link, and that distinction is the whole point.

The rule to apply: loss at one hop that disappears at subsequent hops is a router being polite about its own CPU, and it is fine. Loss that begins at a hop and persists all the way to the destination is a real problem at that hop.

5. If it is the application, what does it actually return?

A connection that succeeds tells you nothing about the response. Request the URL properly and look at the status code, the headers and the body — the common case that defeats simpler checks is a 200 response carrying a rendered error page, which every layer of infrastructure monitoring will report as healthy. If the failure is behind a login or several steps in, no single request will reproduce it and you need something that replays the whole journey.

The check that makes this unnecessary

Every step above reconstructs, under pressure, information that could have been recorded continuously. The reason this playbook is usually run at all is that nobody was watching, and the reason it takes an hour is that there is no baseline to compare against — no record of what latency normally is, when the route last changed, or whether this has happened before.

Tools used in this playbook

For the underlying concepts rather than this procedure, NetTests.NET covers the topic in more depth in its guide.