Acorn Labs
Acorn Labs
Our own record · 20 September 2026
On 20 September we released a fix for a failure that was filling the machine’s temporary disk. The tests for that code passed. All 112. The same morning the fault was back. The tests called the checker the way a comment said to call it. The program that actually runs calls it differently. The safety checks were empty. A file whose name ended in “.lock” was treated as a lock the system was holding.
The suite stayed green. The machine did not. Within minutes of clearing the mess, new copies of the same fault were there, and the temporary disk was filling again. The fix was finding that the tests and the service were not calling the same thing.
The incident was written down that day. A follow-up change was then landed so the tests have to go through the path the service uses. The commit numbers stay in the company record, not on this page.
The tests may no longer invent an input the real program does not send. A lock has to be a lock the system holds. The test run no longer walks into old working copies and treats them as the current product. The landing is recorded. What we can show is smaller. A later run showed those guards switched on. The cleaner that deletes files was left paused.
If you pay for a change in the business, “the tests passed” proves nothing on its own. Someone has to look at the job that really runs, or at the thing the customer sees. We got this wrong on our own tooling. The person who built a change does not mark it correct.