How two servers went months without a backup while every log said everything was fine.
Kludoku runs on a handful of small machines scattered across Europe, North America and Asia. They are not identical and they are not fancy. One of them decides, every minute, what the others should look like: who is serving players, who holds the writable database, who carries a copy of it. That program is a few hundred lines of shell script, and for months it has been doing one thing beautifully — it has been failing without ever saying so.
The specific failure: two of those machines were supposed to be carrying a live copy of the game database, and they never got one. Not once. Meanwhile the console said the fleet was healthy, the logs said the relevant step had run, and the step in question printed nothing at all. This is a story about that, because the fix turned out to be one flag on a command every programmer alive has typed a thousand times.
The manager checks each machine in turn. In shell, that is a loop that reads a list of machines line by line, and for each one runs a command over the network to look at that machine. It reads roughly: for each machine in this list, go and ask it about its database.
Here is the part nobody notices. The tool used to run a command on another machine —
ssh — reads from its own standard input by default and forwards whatever it finds to
the far end. That is deliberate and usually useful: it is how you pipe a file to a remote command.
But inside a loop that is being fed a list, the loop's list is that standard input. So the
first time the loop reaches out to a machine, the command helpfully swallows every remaining line
of the list on its way past.
The loop then ends. Not with an error — it ends the way any loop ends when the list runs out. Politely. Having handled exactly one machine.
The evidence had been sitting in our logs for months and read as good news: six machines registered, four of them meant to carry a database, and three lines of output. The first two machines were skipped for legitimate reasons before any network call happened, so they were logged. The third was contacted — and after it, silence. The two machines that most needed attention were never skipped, never refused and never mentioned. They were eaten.
The fix is the flag -n, which tells the command to read nothing. It is one
character of meaning. It has existed for as long as the tool has.
This is the part worth dwelling on, because the bug is trivia and the reason it survived is not. Every individual piece of that system was tested. The loop was tested. The check it runs was tested. What was not tested was the loop containing the check, on a list longer than one, which is the only arrangement in which the bug exists.
Worse, the system was built to be reassuring. When a machine could not be reached, the code said — correctly, and with good intentions — that not being able to see a machine is different from knowing it has no database, and that it should therefore change nothing. Sound reasoning. But combined with a loop that ended early, it meant the honest answer "I could not see it" and the catastrophic answer "I never looked" produced exactly the same silence.
Once we started pulling, it kept coming. A version of the deploy program refused to install a release that had previously failed on that machine — a good rule — and then exited without ever checking whether anything was still running there. Refusing a new version and refusing to run the installed one are different decisions, and only the first had been intended. One machine sat for a while with a database and a web server and no game on it, and its log read like ordinary housekeeping the entire time.
A cleanup task that removes a database nobody is using ran every minute forever, because the tool it used to check for that database answers for containers that merely exist, not only ones that are running. So it "stopped" an already-stopped thing sixty times an hour and explained itself at length each time. A log line that fires when nothing happened trains you to stop reading the log.
And the one that stung: our own release process was quietly removing servers from the world's address book. The system correctly declines to send players to a machine running an old build. That judgement had been wired into the list of machines eligible to be a public front door — and anything not on that list gets its address deleted. So every time we shipped, any machine that had not finished updating lost its door. On machines that could not update at all, that deletion was permanent. We were doing it to ourselves, several times an evening, while looking for an external cause.
Not the individual bugs. Those are one-line fixes and they are boring. What changed is what the machinery is allowed to do quietly.
A step that decides to do nothing now says which machines it considered and why it passed on each. A step that attempts something reports what happened and then goes and looks, rather than trusting the exit code of the thing it just ran. A failure is classified — this machine's container runtime cannot start anything, that one's disk is full — and printed with the one command that would prove it, because "it failed" and "it failed for a reason you can fix in thirty seconds" should not look the same. A machine that is a little out of date keeps its front door and loses only its share of new traffic, because "worse" and "dead" are not the same word. And each machine now reconciles its own database against what the fleet says it should have, using the connection it already holds, instead of waiting for a distant manager to reach it over a network path that might be the broken thing.
More than it looks like. Kludoku's whole promise is that a case can be solved by reasoning alone — that you are never asked to guess, because every board is proved to have exactly one solution before anyone sees it. A system that reports "everything is fine" when it has not checked is asking you to guess. It is the same offence, committed by us instead of by a puzzle.
The uncomfortable part is that nothing was lost. Two machines were missing their backup for months and the day it mattered never came. That is not a happy ending, it is a near miss, and the only honest thing to do with a near miss is write it down.
Today's cases · More devlog · About Kludoku
Published · More from the devlog