Acorn Labs
Our own record, still being checked
On 18 September a checker’s scratch script tried to make a temporary folder and failed to change into it. The temporary disk was full, so the folder was never created. The script did not stop. It then committed tens of thousands of files into the live code. The files that were already the product were not rewritten.
It was repaired the same morning.
It was written down that morning. The live code was put back the same morning. The damage was files added on top. A list of what to ignore had been overwritten, and one stray file had been added. The repair removed that and restored the list. The bad commit was kept as evidence, not left in place.
The rule, the same morning: a failed step must stop the script. If a change of directory does not happen, that is a failure and the script exits. It never carries on in whatever folder it happens to be in. Write commands only run from a folder it has just created and checked.
The folder a tool is sitting in is the real thing. When a step fails quietly, the next command hits whatever is in front of it. This happened with our own checker, on our own code.