Four Things I Didn't Believe in March

Six months ago, if you'd told me a working prototype and a product people actually rely on were basically the same distance apart, I'd have agreed with you. I don't anymore. Here's what actually changed my mind, one shipped mistake at a time.

Prototypes are easy. Dependable is hard, and none of that difficulty shows up in the demo.

The gap between "it worked when I clicked through it" and "it's still correct after someone else clicks through it a hundred different ways" turned out to be most of the actual work, every single time. The core engine of one testing tool I built came together fast. The audit that later found two systemic bugs hiding under fifty symptoms took considerably longer than building the feature in the first place — and that audit is the part nobody would ever put in a demo.

A good AI product is a workflow with a model inside it, not a prompt with a UI wrapped around it.

Where the model sits in the process, and exactly what happens the moment it's wrong, matters more than anything about the prompt itself. I learned this the annoying way, watching a rebuilt feature silently fall back to generic templated text when its real AI call failed — and say nothing about it. The fix wasn't a better prompt. It was making the failure visible instead of letting it hide behind confident-looking output.

Anything can be made to look impressive once. A fixed bar, applied honestly, is what turns a demo into a capability.

I trained six versions of a private language model this year. Two of them looked genuinely good in casual testing and still failed the evaluation bar I'd set before I ever saw their output — so they didn't ship. That's the whole discipline: deciding what "good enough" means before you're emotionally invested in a specific answer, not after.

Deciding what "good enough" means before you're emotionally invested in a specific answer — not after.

Governance has to be architecture. Bolted on afterward, it turns into paperwork nobody follows.

I found this one the hard way, sitting with my own hiring tool and realizing a scoring signal I'd added without thinking now lived permanently inside a ranking a human would trust. Pulling it out after the fact was possible, but it was surgery. It would have been a five-minute decision if I'd asked the right question before writing the first line of the ranking logic instead of after.

None of these are things I could have just read and absorbed. I'm fairly sure I did read versions of all four, at some point, in some governance document or engineering blog, and nodded along without them actually landing. What changed wasn't the information. It was building something, watching it fail in a way that was specifically my fault, and having nowhere to hide from that.

What changed wasn't the information. It was building something and having nowhere to hide from the failure.

By the time I looked back at the actual record — every conversation, every build session — I'd worked across three different AI tools and somewhere north of a million and a half words just talking through what to build and why. I don't say that as a badge. I say it because if there's one habit worth stealing from all of this, it's that one: get a second, independent read on your own work, from a different vantage point than the one that built it. I did that by switching AI tools mid-project more often than was probably efficient. It caught things a single tool, agreeing with itself, never would have.