A deploy of office-platform failed with an exit code and almost no output: 450 characters, of which the most useful was a warning about .npmrc.
The exit code was 137. That is 128 + 9, and it means the process was killed — not that it errored. Which also means there was no stack trace, no exception, and nothing in the application log explaining itself. The application did not fail. It was stopped.
Following it to the host
The kernel log had the answer:
`` Memory cgroup out of memory: Killed process (node) ``
Then docker inspect on the workspace container, which is where the actual finding was. Memory limit: 536870912 bytes. One half of one gigabyte. And 0.25 of a CPU.
The host had 9.2 GB free.
The cause was not a misconfiguration in the ordinary sense — nobody had typed 512 into a file. Our resolver falls back to a free-tier default when a member of the workspace declares no resources, and no member declared any. Every app got the default, and the default was smaller than the build.
Declaring the real shape fixed it: ten web apps at 384 Mi and 0.3 CPU, three servers at 256 Mi and 0.2 CPU, for a container cap of 4.5 Gi and 3.6 CPU. After that, thirteen processes online, zero restarts, and the site answering 200.
Two smaller findings behind it
The same deploy surfaced a second problem that only shows up on a phone: ten of thirteen CNAME records failed to create, because the DNS provider was over its record quota (81045 Record quota exceeded). The apps were fine; the names were not, which means the fix for that one is not in code at all.
And one native dependency — a media worker — failed its postinstall because the prebuilt binary was built for glibc and the image is musl. That is a 127 from a missing loader, and it looks exactly like the file not being there.
What we changed
The container limit is now a declared number, not a fallback. More generally: a default that is only ever exercised by a failing build is not a default, it is an outage waiting for a workload bigger than itself. And an exit code of 137 should be read as a sentence — the process was killed — not as an error message that happened to be missing.
The cost of not knowing this is what it looked like from outside: a deploy that fails, reports nothing, and retries into the same wall.