On 9 September our package registry went down twice. The first outage had one cause and a clean explanation. The second was caused by the fix.
The first outage
A server-side deploy composed an app's environment from three sources and never called the resolver that supplies credentials. The function it did call is deliberately careful — it emits only non-secret identifiers, on the documented understanding that secrets come from the vault resolver. Nothing on that path invoked it.
So a server-side deploy emitted no database URL and no storage credentials, for every app, structurally, since the path was written. This time it hit the registry: a container recreated without a database URL, crash-looping — while the platform reported the deploy as deployed and the health endpoint returned ok.
Two things about that are worse than the outage. The deploy reported success, and the health check agreed. Neither of them had ever been asked "does this app have the configuration it needs to start?" Half a day went into chasing a composer regression, including in a file that turned out to have no production callers at all — several of us named it as the cause because it looked like the mechanism, and it was not on the path.
The two hours in the middle
While the registry was down, something else was happening that should have been found in minutes rather than hours: the registry was serving an unrelated application's website at HTTP 200. Including on tarball URLs. Which meant package managers were downloading an HTML page and reporting a tarball integrity error — so every repository in the fleet started debugging its own lockfile.
A second deploy fault had left a generated proxy route uninstalled, so a three-day-stale route was still pointing at a port that another app had since taken.
Four things that failed open
The registry's own postmortem lists them, and the list is the most valuable artifact of the day:
- The misroute answered 200 with another app's HTML. A wrong answer that looks like a right answer.
- Deploy verification asserted a 2xx status — which the misroute satisfied.
- The ping endpoint reported ok while the health endpoint reported degraded.
- A missing session secret generated a random one and logged a warning. So a misconfigured instance started successfully and behaved unpredictably.
Every one of those mechanisms was built to avoid failing. Together they turned a two-minute diagnosis into a two-hour one, and they are the reason it took a developer's complaint to notice rather than a monitor.
The fix that was worse
The second outage is the part worth the post. The fix emitted a database URL that was present and wrong: correct host, correct port, correct database, correct role, correct credential — missing ?sslmode=disable.
The driver's default is to require TLS. The server it was connecting to has TLS disabled. So every deploy crash-looped.
Replacing "missing" with "present and wrong" was worse than the original fault.
A missing value fails immediately and obviously. A wrong value fails the same way but looks like progress, and it consumed a second investigation on the same day, in the same system, by people who had just spent a day on the first one.
One more finding, worth its own paragraph
While untangling it, we found that five deploys were becoming up to fifteen renders. Our supersede logic guards on a timestamp column being null, and that column carries a default that fills it in — so the guard has never once been true, and supersession has never fired in production. Separately, the queue sets three attempts while the worker's own comment says no attempts are configured.
What we changed
Deploy verification now asserts what the app can do, not which status code came back. Health checks can fail rather than invent a secret. And the rule we wrote down for ourselves: when a deploy fails because a value is absent, the fix must be verified as a value, not as a presence — because "it is set now" and "it is right" are different claims, and the second one is the only one that works.