AI Inside Your Product: From Demo to a Feature That Survives Production
What separates a model that works in a demo from a feature thousands rely on: context, guardrails, cost, and designing for failure.
Building an AI demo is a day's work now. Building a feature thousands of people depend on daily is a different exercise entirely — and the difference is not the model, it is everything around it.
The problem is context, not intelligence
The model does not know your customer's data. "Write a description for this product" with no context produces text that fits any product, which means it fits none. Real value comes from injecting the right context: the customer's own content, their audience, their existing tone. That is retrieval engineering, not prompt engineering — and the original retrieval-augmented generation paper is where the idea comes from.
Retrieval before generation
- Chunk content on meaningful boundaries — lessons and sections, not every 500 characters.
- Retrieve only what the request needs; stuffing the window degrades quality and raises cost simultaneously.
- Attach the source to every result, because users need to verify rather than trust.
- Scope retrieval by tenant with the same rigour as your database — a leak through context is still a leak.
Guardrails are not a word filter
The real guardrail is architectural: define what the feature can read and what it can execute. A model that suggests text is orders of magnitude safer than one that publishes a course or emails a thousand learners. Route consequential actions through explicit human approval rather than through confidence in the output.
Do not ask "is the model good enough?" Ask "what is the worst thing that happens when it is wrong?" The second answer determines the design.
Cost compounds quietly
In a demo cost is a rounding error; in production it is a line item. Control it three ways: cache repeated requests, size the model to the difficulty of the task rather than always reaching for the largest, and cap usage per tenant so one customer cannot consume your margin.
Design for failure, because it will happen
The model will be slow, will occasionally return something unparseable, and the provider will have an outage. Treat every call as a network request that can fail: explicit timeout, bounded retry, and a fallback path that keeps the rest of the feature working. An AI feature that takes the page down with it is worse than no feature.
Make evaluation part of building
"Looks good" is not a test. Collect a set of real inputs with acceptable outputs and run it on every prompt or model change. Without it you will not notice that a small edit broke a case that used to work.
For designing these features, Anthropic's guide to building effective agents draws the clearest line between what warrants an agent and what a single call already handles. For a governance framework you can put in front of an enterprise buyer, NIST's AI Risk Management Framework is the accepted reference.
If you are planning an AI feature in your own product, the hard part is deciding where it belongs, not how to build it. Talk to us.
FAQ
Should I fine-tune my own model?
Rarely justified. Good retrieval with a general model beats fine-tuning for most product use cases, at a fraction of the cost.
How do I stop the model inventing facts?
You do not eliminate it. Constrain it to retrieved context, show the source, and gate sensitive actions behind approval.
When is a smaller model the right call?
Classification, extraction, and short summarization — most in-product tasks do not need the largest model available.