When we demoed Q.Promo AI, the question everyone asked was about the model. Which LLM? How did we prompt it? Nobody asked about the part of the system that actually made it trustworthy: the path that doesn’t use the model at all.
Q.Promo AI answers promotion questions for retail analysts. About 70% of those questions are ones the system has seen before, in some form. For those, we don’t generate SQL. We route to a pre-validated playbook — a query a human has verified — fill in the parameters, and return the answer in 10–12 seconds. Guaranteed correct, every time.
95% accurate is 5% wrong
Our generative path — the one that writes and verifies its own SQL — runs at 95%+ accuracy. That number is genuinely impressive, and it is nowhere near good enough on its own. A category manager reallocating a promotion budget doesn’t experience “95% accurate.” They experience one answer. If it’s the wrong one, the system’s credibility is gone, and it doesn’t come back.
The first job of the system is to know what it doesn’t know. Certainty where certainty is possible; reasoning only where it’s necessary.
The system gets more deterministic over time
The part I’d keep in every future system: when the reasoning path answers a novel question correctly, we save it as a new playbook. Tomorrow, that question is an instant answer. The generative path is how the deterministic path grows — the model is a mechanism for manufacturing certainty, not a replacement for it.
If you’re building a copilot over your own data, my advice is one sentence: spend as much engineering on when not to generate as on generation itself.