A postmortem: about seventy minutes of downtime, one import line, and three fixes.
On the evening of 18 August 2026, Kludoku went down. Every address — the main site and the regional doors — answered with an error for roughly seventy minutes. For part of that window it was worse than an error: some visitors were handed a small file to download instead of a web page, which is precisely the kind of thing that makes a person close the tab and never come back. Nobody was attacked, no account was touched, and no puzzle data was lost. It was our own mistake, twice over, and this is the whole of it.
We had just shipped the biggest content update in the game's short life: illustrated guides, written recaps of each day's cases, and those recaps translated into all fifteen languages the game speaks. One small ambition inside that work caused all the trouble. When a recap needs to say chef, or The Old Observatory, or inheritance, it should use the exact word the game already shows you in your language — never a second, slightly different translation invented by the part of the code that writes the article. So the recap module reached across and read the game's own translation files.
In the source tree, that works perfectly. In the thing we actually ship, it cannot: the server package deliberately does not carry the player-facing source folder, because the playable app is already built and baked in elsewhere. And the language the server runs in resolves every import before it runs a single line of the program. So the server did not start, stumble, and recover — it never started at all. Each machine tried, failed instantly, and tried again.
Every automated test passed, because tests run against the source tree, where that folder exists. The tests were answering a question we had not thought to ask: not "does this code work?" but "does the thing we are about to ship work?" Those are different questions, and only the second one matters at three minutes past a release.
With the real front doors unable to serve, traffic fell through to a back-office machine that normally answers nothing but a one-line note explaining it is not the front door. That note went out with no statement of what kind of content it was — and a browser given content it cannot identify does the only safe thing it knows: it saves it to disk. So the site appeared to be handing out mystery files. It was, in the most boring possible sense, a note that said "wrong door", missing its label.
Here is the part that stung. A corrected release was published minutes after the cause was found. The machines could not take it.
Each server runs a small updater on a timer: it does a few housekeeping chores, then checks whether a new release exists, then installs it. One of those chores asked whether the game's container was present before talking to it. A container that starts and dies over and over is very much present — so the chore proceeded, tried to talk to a program that was not running, and failed. The updater is written to stop at the first sign of trouble, which is usually the right instinct and was exactly wrong here: it stopped every minute, at the chore, before ever reaching the question "is there a fixed version waiting for me?" The cure was sitting in the registry the whole time, one step past a door that kept slamming.
We now boot what we ship. Before a release goes out, we assemble the published package exactly the way the server does — nothing from the source tree, only what actually travels — start it, and confirm it reports the new version. It takes twenty seconds and it would have caught this outright.
A release that cannot start is undone automatically. Each machine now remembers the last build that proved itself by answering a health check. If a new one never answers, the machine returns to the proven build, marks the bad version as one it will not install again, and says so in the console — so a broken release becomes a blip on one machine instead of an outage on all of them.
The machines rescue each other. Chores can no longer block the update that fixes them, and a server that has gone quiet for ten minutes now gets a knock on the door from the machine that coordinates the fleet: clear the wedged state, install the current release, rejoin. The system's ability to heal itself should not depend on someone being awake.
The database was untouched — the failure happened before any of our code ran, which is a small mercy of how the language loads programs. The day's cases are generated from the date itself, so nothing had to be recreated: when the servers came back, the same cases were exactly where they had been, with every solved board and streak intact.
We would rather write about crime scenes than about our own. But a devlog that only publishes the good days is marketing, and the last entry here ended with a line we apparently needed to take more seriously: any machinery that can fail silently eventually will — make it speak. This time it spoke. It just took us seventy minutes to be standing in the right room to hear it.
Today's cases · More devlog · About Kludoku
Published · More from the devlog