In October we found out that several things we believed were running were not. Here are nine lessons. Each says what happened, what we changed, and what our own record shows.
1. No lead had ever reached the lead system, and nothing said so
On 13 October we started listing places where something could fail without telling anyone. The list now holds 78, and 31 are closed for certain. One example: lead forwarding checked only for network errors and returned quietly when its key was empty. The leads file had been 0 bytes since 8 September, and no SheetTamer lead had ever reached the lead system. We rebuilt the forwarding so a missing key is logged once, a bad response is logged, and a server error is retried. It passed review on 31 October with a test for each path.
2. Our own deploy script was one of the silent ones
A ship stopped at a host test step with exit code 1 and no output. The cause was a shell assignment that captured the remote command and its exit code under a setting that quits on error, so the script exited before it printed anything. Every failing remote step had been silent. We changed the line, and added a test that passes on the fix and fails on the old line. The suite then passed 580 of 580.
3. Our tests were checking a build from four days earlier
On 24 October the browser journeys ran four times in one night against the live image, which had been built on 20 October. Every pass and every fail described old code. What gave it away: a journey that needed a rule added on 23 October could not find it. The rule we kept: a test that runs a built artifact first asserts that the artifact carries the commit under test. The runner now refuses to start on a preview image that lacks that rule.
4. One test database cannot have two users
On 23 October two runs shared one test database. One dropped it under the other, which produced 84 false failures in one run and 5 in the other, and cost us a ship. Parallel work can split code. It cannot split a shared resource. Now any brief that allows a full test run names the window.
5. A newer model changed the shape of the answer
We tested a newer model tier on the receipt reader with 24 synthetic receipts, 144 fields. The current model read all 144 correctly. The newer one scored 125 as our code read the reply, with three parse failures. The answers were there, but our code read the wrong block. The reply began with an empty reasoning block and we read only the first block. On a rerun with a larger output limit, 5 of 24 replies began that way. We changed the reader to take the first block that holds text, and the newer model then scored 143 of 144. The one real miss was a payment method written as a check number. Lesson: a model swap tests your parser as much as the model. And one run of 24 clean synthetic receipts says little about real phone photos.
6. A tax tips video was wrong, and our tax review caught it
Our review panel screens every video we are sent before it can shape the product. It screened a video listing small business tax write offs. Our tax review found it wrong on the 2026 mileage rate and incomplete on home office, health insurance and meals. Our conclusion: a tax claim from any video goes to our tax review before it can shape a feature or a word of copy. We wrote five of its topics into the backlog as questions to answer properly, and moved meals ahead of launch.
7. We turned down a popular plugin bundle. Here is why
A popular bundle of coding plugins came through the panel in October. Our skeptic reviewer stopped it: the bundle was unpinned, its 27 default hooks logged tool output in plain text, and it added about 31,000 tokens to every session. We did not install it. The panel logged 32 entries that month. 11 led to a recorded change and 21 changed nothing, which is the point of screening.
8. Our most capable model did too much of the work
Our goal is for the top model tier to do 60 percent of our weighted model use. In one 16 hour stretch, 23 to 24 October, it ran 71 percent. The main session alone was 42 percent: 1,354 calls, 703 of them carrying over 400,000 tokens of context, because that session ran past our refresh line. We started this method in the middle of the month, and we track it from here. The rule now: at 200,000 tokens the session writes a state file and offers a fresh start, and lookups go to a cheaper agent that returns only the verdict.
9. Written, merged and running are three different states
We wrote a retention schedule for SheetTamer in four steps: close and purge clocks, redaction guards, an audit log and suppression list, and indexes for the clocks. Then we corrected the document so it says the jobs are built but not yet running in production. Written, merged and running are three different states, and a public promise should name only the last.
None of this is glamorous. Most of it is checking whether the thing we believed was running is the thing that ran. If you want that check run on your system, tell us what you think is running.
Want this discipline in your build?
We find the failures that say nothing.
Silent failure audits, tests that prove the right build ran, and agent systems with human review gates.
