Start with the decision, not the model
Write down the decision the user is trying to make, the evidence they need and what happens when the answer is wrong. If the answer cannot be checked or the cost of an error is too high, a different interface — or a human process — may be the right solution.
Retrieval before generation
Most business AI problems are retrieval problems: the right policy, product note or historical case is not in front of the person. Retrieval-augmented generation can help only when the source, version and permission model are clear. Better retrieval usually improves answers more than changing the model.
- Which sources are approved, current and owned by someone?
- Can the system show the passage that supports the answer?
- Does the user have permission to see that passage?
- What should happen when no source is found?
Evaluation is part of the product
An evaluation set needs questions with known answers, ambiguous questions and questions with no answer in the corpus. Measure retrieval separately from generation, then review a sample by hand. A system that says “I do not have enough evidence” is often more useful than one that always answers.
Access control and data boundaries
Enterprise retrieval must filter by role, source and tenant before content reaches the model. Logging, retention and deletion need an owner. A demo that works on twenty documents does not prove that the system can handle documents that change daily or that different teams must not see.
Cost, latency and failure
- Measure cost per request at realistic context sizes, not only in a prototype.
- Set a latency budget and a fallback when the model is unavailable.
- Track the rate of answers that require human review.
- Record failures by type so the data, retrieval or prompt can be improved.
See AI integration services, what drives custom software cost or discuss your project.