Method document
Evaluation method
and report outline.
How EvalAnchor reviews a RAG assistant: from agreeing scope to a finding your team can reproduce, with the evidence behind every step.
01Purpose
This document describes the method behind an EvalAnchor evaluation and the structure of the report a pilot produces. The launch kit demonstrates traceable RAG evaluation using original, fictional product policies: 30 questions and 60 authored fixture responses. No model API calls were made to create it.
The goal is not a single score. It is a set of findings that show where an assistant fails, why, and what evidence proves it.
02Five-stage workflow
Every engagement follows the same connected stages, so each finding can be traced back to the question and evidence that produced it.
Agree on scope
Use cases, authorized documents, review criteria, and data-handling terms are agreed before any material is shared.
OutputScope recordCreate representative questions
Everyday requests, edge cases, and questions whose answer is absent from the documents, each with a reference answer.
OutputCase set with referencesInspect retrieval
Did the system find the right documents, and select passages that actually address the question?
OutputRetrieved vs. relevant evidenceReview answers and citations
Each important claim is checked against the passage it cites.
OutputClaim-level judgmentsClassify failures
Human reviewers assign a failure category and record their reasoning.
OutputReviewed findings
03Review dimensions
| Dimension | Reviewer question |
|---|---|
| Answer correctness | Does the response meet the reference answer and expected behavior? |
| Evidence support | Do the cited passages support the response’s claims? |
| Citation validity | Are citations resolvable, relevant, and attached to the right claim? |
For every judgment, reviewers record the evidence they relied on, their reasoning, any critical defect, and time spent.
04Failure categories
Missed document
A relevant source existed but was never retrieved, so the answer could not use it.
Irrelevant passage
A retrieved passage does not address the question, yet shapes the answer.
Unsupported claim
A claim in the answer is not backed by any retrieved or cited evidence.
Citation mismatch
The citation points to a passage that does not support the claim it is attached to.
05Human review and adjudication
Reviewer A
Scores independently, records evidence and reasoning.
Reviewer B
Scores the same response without seeing A’s judgment.
Adjudication
Disagreements and “uncertain” judgments are resolved explicitly before a case is accepted.
Automated checks verify file integrity and evidence references. They cannot approve a case.
06Example finding record
- Question
- How long do customers have to request a refund?
- Retrieved passage
- Shipping policy · delivery times
- Generated answer
- “Sixty days. [1]”
- Citation
- [1] → Shipping policy
- Failure category
- Unsupported claim · Citation mismatch
- Reviewer note
- The refund policy (not retrieved) states thirty days. The cited passage does not mention refunds.
07Report structure
- Corpus and configurationVersions of documents and system under test
- Case inventoryAll cases, with exclusions and reasons
- Metrics by system and splitNumerators and denominators shown separately
- Failure examplesEach linked to the exact passages involved
- Disagreements and resolutionsHow reviewer conflicts were settled
- Limitations and reproductionSteps to rerun the check yourself
08Limits of this method
- Source-ID checks do not prove semantic support; that requires human judgment.
- Authored fixtures are not a benchmark of any real model.
- A small, purposive case set does not estimate production reliability.
- Performance claims require human review and a run against a real, frozen system.