Forge — It Wasn't Fifty Bugs. It Was Two.

My team's API test automation had been stuck for years — individual endpoint scripts, no real journey coverage, nothing a business stakeholder could actually read. I didn't just want to audit that problem. I wanted to build a working alternative myself and see if it held up.

The corporate reality made the first decision for me. I couldn't guarantee Python or admin rights on every machine, and anything that looked like a .bat or .exe launcher got flagged by antivirus before it even ran. So no Streamlit, no fancy dependency tree — just Flask and vanilla JavaScript, install friction close to zero. I also killed the idea of an always-on LLM in the core flow almost immediately. A live API call needed a functional ID that needed approvals, and it kept hitting CORS walls on the network.

The tool had to work without asking anyone's permission first, so I built a fully offline, schema-driven rule engine instead and kept AI as something you could switch on later, not something you depended on from day one.

That part went fine. The part that nearly broke me was much smaller-sounding: cards kept flipping from ACTIVE to BLOCKED forty-five times inside a single regression report.

I described the cycle to one of the AI tools helping me build it as "tom-and-jerry" — fix one thing, break another, chase it around the file, over and over.

For weeks I treated every failure as its own bug. Wrong card status, fix it. Assertion pointing at the wrong record, fix it. A negative test case somehow expecting a success response, fix it. Each fix held for a day, then something else broke in a slightly different shape. I kept assuming I just hadn't found the right patch yet.

What actually changed things was stepping back and auditing eight full runs at once instead of chasing the ninth failure. And the pattern that fell out of that audit was almost embarrassing in how simple it was: the tool didn't have fifty bugs. It had two. Somewhere in the codebase, fourteen separate functions — spread across Python and JavaScript — each had their own opinion about what a test's "expected status" should be, and none of them talked to each other. On top of that, five different mechanisms were all trying to manage test data state at the same time. Every "new" bug I'd been fixing for weeks was just two systemic disagreements wearing a different mask each time.

I scrapped the idea of patching my way out. Instead I built what I ended up calling a Spec Oracle — one function, and only one, gets to decide what a test's expected outcome is. Everything else has to ask it. Data provisioning got its own single gatekeeper the same way. And I stopped pretending every test result was a clean pass or fail — some things aren't wrong, they're just uncertain, so results now land in green, amber, or red instead of a binary that hides the difference.

There's a smaller story inside this one that I still think about. Partway through, I asked an AI tool to generate the UI mockups, and it got stuck in its own loop — missing browser binaries, broken renders, retry after retry. I opened a completely different tool, made better mockups myself in a few minutes, and told the AI to follow those instead of trying again. It's a strange thing to notice about yourself: that you're faster at a task than the tool you're using to speed everything else up. I don't think that means the tool is bad. I think it means treating AI output as a first draft, always, is the only version of this that actually works.

Every "new" bug was just two systemic disagreements wearing a different mask each time.

The tool is called Forge now, and it's grown well past testing into something closer to an engineering intelligence layer — failure investigation, institutional memory, the kind of thing that used to live only in a senior engineer's head. But the part I'm actually proud of isn't the size it grew to. It's noticing, eventually, that a pile of unrelated-looking bugs is sometimes just one bad architectural decision, repeated.