SERVICE / RAG EVALUATION
Make your next release
an informed decision.
A focused evaluation pack for a documentation or support assistant, with traceable evidence and a human review process.
At a glance
| Who it is for | Teams running a documentation, support, or internal knowledge assistant that answers from their own documents. |
|---|---|
| You provide | Authorized documents, one user workflow, and two frozen sets of your assistant’s answers. |
| We do | Write 50–100 evaluation cases from your documents; two reviewers judge every answer for correctness, evidence support, and citation validity. |
| You receive | Reviewed cases (CSV + JSONL), a failure report with evidence, reproduction instructions, and one correction cycle. |
| Timeline | About three weeks after scope, access, and the rubric are agreed. |
| Price | Quoted after we review your corpus and acceptance criteria. |
See the three checks in the interactive demo →
One corpus. A clear scope.
The proposed pilot covers 50–100 accepted evaluation cases and two frozen response sets per case. We assess each response for answer correctness, evidence support, and citation validity separately.
Why three separate checks
A response can be well-written and still fail one of these checks alone — that is exactly what a single accuracy score hides. Here is one fictional example carried through all three.
Illustrative example. Question: can I return a final-sale item. AI answer: yes, all items can be returned within 30 days, citing source 1. Answer correctness fails, because the answer contradicts the final-sale exception. Evidence support fails, because the cited passage never mentions final-sale items. Citation validity passes, because citation 1 resolves to a real, existing section. Case verdict: one of three checks pass; an unsupported answer was found and escalated for review.
What your team receives
- 50–100 questions with expected behavior, exact source passages, and case categories.
- Two frozen responses per question, with independent judgments and documented resolutions.
- A failure analysis, case-level CSV and JSONL, and reproduction instructions.
- One correction cycle for defects against the agreed rubric.
How we review
An evaluator authors and assesses the cases. A separate reviewer evaluates every case and response. We retain both judgments, resolve disagreements explicitly, and exclude unresolved records from quality claims. Automated checks support the workflow; they do not replace human judgment.
Plan for three weeks
The delivery target begins after scope, authorized inputs, access, and the calibration rubric are agreed. Week one establishes the reference set; week two covers frozen outputs and review; week three resolves findings and prepares the handoff. Final commercial terms are agreed with you.
Useful questions
Which systems are a good fit?
Documentation and support assistants with an authorized, versioned corpus and a defined user workflow. We agree answerability and escalation rules during calibration.
Can you compare two versions?
Yes. The pilot includes two frozen response sets on the same questions. Model, prompt, retrieval settings, and corpus versions must be recorded before the holdout is run.
Do you provide a universal accuracy score?
No. Results depend on the selected cases, rubric, and system configuration. We report the three dimensions separately with denominators and limitations.
What is excluded?
Model training, a production monitoring platform, security certification, unrestricted red teaming, and additional response sets are outside the standard pilot.
How much does a pilot cost?
We quote after reviewing the corpus, case complexity, access constraints, and acceptance criteria. Contact us to scope a pilot.
Follow one case end to end
An illustrative RAG case, from agreeing the scope to a reviewed finding.
- 01
Choose what to check
We agree on your documents and what a good answer means.
Agreed scope - 02
Ask real questions
We prepare questions your customers might actually ask.
Questions + reference answers - 03
Collect AI answers
We save what your AI says and the sources it uses.
Illustrative exampleWithin 30 days of delivery. [1]
Answers saved with their sources - 04
Check answers & sources
Is the answer right? Does the cited source support it? Retrieval checks require available traces.
Return it within 30 days of delivery. [1]
[1] Example return policy · §2A citation points to the source behind an answer.“Returns are accepted within 30 days of delivery.”
Right answer?Supported claim?Right source? - 05
Review & Resolve
Reviewers check independently, using the same rules. Resolve disagreements and record final reasons.
Final decisions, with reasons - 06
Get clear findings
You see the problems, the proof and what to investigate next.
Clear report + reviewed examples