Builder notes on the AI tools I run on real deals.
One idea per post. Architecture decisions, things I tried first and abandoned, what I'd do differently.
- 016
Deployed is not redeployable
A service can run healthy for weeks while quietly losing the ability to come back. None of my monitoring asked the second question.
AI2026 · 08 · 22 - 015
The deploy was live. My browser was running last week's code.
The server said the new version shipped. The page in front of me disagreed. It took a while to learn they were both right.
AI2026 · 08 · 19 - 014
The container ran fine. The folder it read was empty.
Getting Docker running without a system-level install was the easy part. The failure it produced was the quiet kind.
AI2026 · 08 · 16 - 013
The failure a human can't miss and an agent can't see
Excel quietly grew to six gigabytes on my Mac. I'd have noticed in a minute. The agents driving it never would.
AI2026 · 08 · 13 - 012
Two copies of one behavior will drift apart
My analysis scripts exist twice — a Windows version and a Mac twin. A bug fixed in one stayed alive in the other for seven months.
AI2026 · 08 · 10 - 011
The agent deleted work that wasn't its own
Two model sessions, one shared folder, and a cleanup command that threw away a morning of edits.
AI2026 · 08 · 07 - 010
A fix isn't proven by a test that never failed
The regression test I trust is the one I watched fail first. The other kind has lied to me twice.
AI2026 · 08 · 04 - 009
The reviewer can't be the author
A postal-code bug sailed past the model that wrote it and got caught in minutes by a model that didn't.
AI2026 · 08 · 01 - 008
One model plans, another builds, a third reviews
The reliability math that keeps my pipeline to one LLM call per document also applies to the process that builds the pipeline.
AI2026 · 07 · 29 - 007
The dashboard was wrong for two weeks. Every health check was green.
Model evals don't catch a broken data layer. The fix is the same discipline pointed at the database — checks on the numbers people actually act on.
AI2026 · 06 · 10 - 006
The feature that tested seven points better and changed almost nothing
How the eval caught a leaked metric — and then told me how to make the feature actually useful.
AI2026 · 06 · 08 - 005
An accuracy number is meaningless without the cost of being wrong
Not all classification errors cost the same. The eval reports two numbers and ranks every failure by business impact.
AI2026 · 06 · 05 - 004
You can't grade a language model against its own guesses
The fastest way to build a useless eval is to use the model's own output as the answer key.
AI2026 · 06 · 02 - 003
The one place the LLM lives in my data pipeline
Dozens of Python tools, exactly one LLM call per document. The boundary is the design decision.
AI2026 · 05 · 25 - 002
Why I moved my rent-benchmark and income-statement workflows out of Claude chat into Skills
Three problems running spreadsheet workflows in Claude chat that pushed me to build dedicated skills.
AI2026 · 05 · 22 - 001
What this site is
An introduction to the site — who I am, what I build, and what to expect here.
2026 · 05 · 17