On 26 August, forge-git refused every clone, fetch and push for seventy-five minutes. All 110 repositories. The first notification we got was a developer whose push failed.
What was not broken
That is the part worth writing about, and it is not the outage.
Our container healthcheck fetched /. Prometheus scraped the node exporter. Both were green throughout. A service that answers its own web UI is not the same service as one that can serve git — and we had only ever watched the half that was working.
Gitea's web UI is served from Postgres. Postgres was healthy. The git backend was not. Every dashboard we had was measuring one of those two things and concluding, correctly and uselessly, that the pieces it could see were fine.
The fix is a better question
The replacement probe does not ask whether the service is up. It asks whether it can do the thing it exists to do: a blackbox exporter requests the authenticated advertise-refs endpoint and asserts a 200. If the git backend cannot serve refs, the probe fails, regardless of what the UI says.
The first version of that probe asserted 401 unauthenticated — on the theory that refusing an anonymous request proves the endpoint is alive. It was wrong, and it would have been wrong in the worst direction: any deployment that accidentally permitted anonymous access would have looked like an outage, and any proxy that answered anything at all would have looked healthy.
The same lesson from two directions
Two other incidents this month were the same idea wearing different clothes.
A queue-depth dashboard reported 342 jobs at status 5 — a number that reads as a jam. It was not. 331 of those rows already carried a task ID, which means they could never be dispatched; the query was counting rows rather than claimable work. After the fix, the same query returns 11, against 13 jobs actually running. The dashboard had been describing its own data model, not the system.
Separately, a set of CI jobs sat in Waiting for hours with task_id=0 and no action_task row at all, while single-label jobs on the same runs dispatched in seconds. The jobs declared two runner labels, and no runner on the fleet carries both — so they were not slow. They had never been eligible.
"The deployment is down" and "we never looked" are not the same statement. Most of what we did in August was teaching our tooling to tell them apart.
What we changed
The probes now ask whether a service can do its job, not whether it is reachable. Scheduled jobs that have never dispatched are alarms in their own right, because a job that is stuck at zero is not a job that is behind. And for anything that reports a count, we now ask what the number would look like if the thing it describes were broken — if the answer is "the same", the metric does not exist yet.