OctoView LabsThe weekend audit
We spent the weekend auditing our own system instead of building on it.
Not really by choice. We asked some agents to go look for problems, they came back with about fifty, and then we couldn't stop reading.
What got us wasn't the number. It was that almost nothing was actually broken. The code was fine. It just wasn't connected to anything.
Best one: every build has been told to run a verification step for the last ten days. We never built it. No binary, no build rule. And the same policy says a rule that can't ever fire is itself a failure, which makes that instruction fail its own rule.
Then there's a watchdog that reports "all hook binaries present." It's pointed at a config file with no hooks in it. It would say exactly the same thing if we deleted every hook we have.
Same watchdog, second check: "agents never ran." There are 65,535 files sitting in the directory it's looking at. It doesn't check subdirectories.
And four of our services have a health endpoint that actually knows whether they're doing work. We run twenty-four health probes. None of them read those four.
This is what happens when things get built faster than they get wired up, and nobody notices, because code that isn't connected doesn't crash. It just agrees with you.
We only found it because we had agents compare what's actually deployed against what's in the source. Without that, all of it looks green.
The ones above are a few hours of work. There's a longer list behind them. The harder part, and what we're doing now, is making the build fail when a check isn't wired to anything.