Most failed GenAI projects I’ve seen didn’t fail on the model. They failed on the architecture wrapped around it — decisions made to look sophisticated in a slide, not to survive contact with production. This is a running list of the patterns I now argue against by default, and the one I argue for.
The rule underneath all of them: add a component only when a specific, named failure forces you to. Complexity you can’t trace to a failure is decoration.
AI-washing
The tell: a workflow that a deterministic rule, a lookup table, or a regular form would handle correctly, dressed up with a model so the project can be called “AI.” You inherit every downside of a probabilistic system — latency, cost, variance, evaluation burden — to solve a problem that was never probabilistic.
Before a model goes anywhere near a workflow, I want an answer to one question: what about this input is genuinely open-ended? If the answer is “nothing, really, the inputs are structured and the rules are known,” the model is there for the narrative, not the task. Cut it.
Where the model earns its place: unstructured input, ambiguous intent, a long tail of cases too large to enumerate. Where it doesn’t: anything you could write an acceptance test for and pass with a switch statement.
Premature fine-tuning
Fine-tuning is where teams reach first and should reach last. It’s expensive to do well, expensive to maintain, and it moves a capability you could have kept in plain sight (a prompt, a retrieved document) into opaque weights nobody can inspect or diff.
The order I actually work in:
- Prompt and instructions. Most “the model can’t do X” turns out to be “we never clearly told it to do X.”
- Retrieval / grounding. If the gap is knowledge, give it the knowledge at inference time. This is inspectable and updates the moment the source updates.
- Tooling and structured output. If the gap is reliability of format or action, constrain the output and give it tools.
- Fine-tuning. Only when you have a stable, evaluated task, a real dataset, and a measured ceiling that the first three can’t lift.
Reaching step 4 first means you’re paying to memorise a target that’s still moving.
TODO — add the on-prem case study where retrieval closed the gap that a proposed fine-tune was budgeted for, with the before/after eval numbers.
Vector-DB theatrics
A vector database is a fine tool for one job: semantic recall over a corpus too large to fit in context. It is not an architecture, and it is not free.
The theatrics I watch for:
- A dedicated vector store stood up for a corpus of a few hundred documents that would fit in context, or that keyword search handles better.
- Similarity search where the real access pattern is exact — a policy ID, a SKU, a customer record. That’s a database query, not an embedding.
- No evaluation of retrieval quality, so nobody notices that the top-k chunks are plausible-looking and wrong.
Retrieval is a search problem before it’s an AI problem. If the retrieval is bad, a better model just writes more fluent nonsense on top of the wrong context. I measure retrieval on its own — recall on a labelled question set — before I let anyone credit or blame the model.
TODO — drop in the chunking-strategy comparison table (fixed-size vs structure-aware) from the regulated-docs build.
Unnecessary multi-agent designs
The multi-agent diagram is seductive because it looks like an org chart, and org charts feel like engineering. In practice, most “agents” I’m handed are a single task split into roles that then spend their budget passing context to each other and compounding each other’s mistakes.
Every hop between agents is a place for error to enter and for latency and cost to multiply. A three-agent pipeline where each step is 90% reliable is not a sophisticated system; it’s a system that’s right about 73% of the time.
My default is one model, one clear task, with tools. I only split into multiple coordinated steps when the sub-tasks are genuinely independent, separately evaluable, and benefit from isolation — not because a mesh of agents demos well.
TODO — write up the case where collapsing four agents into one grounded call cut p95 latency and raised task accuracy.
The boring architecture that actually ships
After all the subtractions, what’s left is usually the same shape, and it’s dull on a slide:
- One capable model, hosted where the data is allowed to live.
- Retrieval only where knowledge is the gap, evaluated as its own component.
- Structured output and a few real tools where reliability of format or action matters.
- A deterministic shell — validation, guardrails, fallbacks — around the probabilistic core, so a bad generation degrades instead of detonating.
- An evaluation set that predates the demo, so “better” is a measurement, not a vibe.
This is boring on purpose. Boring is inspectable, boring is cheap to run, boring is what’s still working in month six when the person who built it has moved on. The job isn’t to build the most interesting system that could work. It’s to build the least interesting one that does.
TODO — add the one-page reference diagram and the deployment checklist I use at kickoff.