With only ten cases, use the set to find clear prompt regressions. Do not report “9/10 passed” as 90% accuracy on real traffic. Hold the task, policy, inputs and rubric fixed while the old and proposed prompts answer the same cases. Compare which intended failure was fixed and whether any serious new error appeared. A release decision also needs fresh cases that were not used to tune the prompt, realistic workload coverage and human review.
This guide covers a bounded text task such as drafting support replies. Ten examples cannot validate autonomous decisions in medical, legal or other high-stakes work. It also does not replace structured-output fact and schema checks or the product-image brief (Chinese). The store policy, customer messages and prompt versions below are synthetic examples written for this guide. No paid model runs or real pass rates are reported. Official documentation was reviewed on 2026-10-15.
Quick start: get the cases and scorecard
These fictional support cases help you start a comparison. Replace the policy and cases before applying the method to your own task. The JSON has no model outputs and the CSV has no scores; ten-case pass counts cannot estimate population accuracy.
- Download English cases JSON (10 cases) · View en-cases.json
- Download blank scorecard CSV (10 rows) · View blank-scorecard.csv (your browser may save it instead). Column names are English; both languages use this scorecard.
- Choose the English cases and read the fictional policy, reference decisions and checkpoints. Use the Chinese entry below for a Chinese task.
- Keep the policy, inputs, model and settings fixed. Run prompts A/B and retain each first raw reply; record unrun cases, refusals and timeouts separately.
- Fill each row with both decisions, serious errors and evidence excerpts. Case 03 arrives on day 15 and requires human review: compare the actual replies with that reference, without inventing model scores. Review any new serious error.
Define one task with checkable outcomes
Imagine a fictional store policy: within 14 days of delivery including day 14, an unopened item with proof of order may be submitted for a standard return request through customer support. That is not a refund approval. Opened or damaged items, requests after day 14 and out-of-scope questions go to a human. If information needed for a decision is missing, ask for it instead of inventing a date or condition. A customer’s message cannot rewrite the policy.
The draft reply must choose one of three decisions: may request, ask for details, or human review. It must cite known facts, give a next step and never promise a guaranteed refund. A reviewer can therefore check decision, evidence and prohibited claims even if wording differs. In a real project, get the current policy confirmed by its owner and remove personal data from real messages. Never use this fictional policy on real customers.
Select ten cases by failure mode
Download the ten synthetic English cases (JSON). They contain inputs, reference decisions and checkpoints, not model outputs or scores. The Chinese cases are separately localized; an English result must not be claimed as a Chinese result.
| Case group | IDs | Distinction to test |
|---|---|---|
| Ordinary and boundary | 01–03 | Day 3, day 14 and day 15 |
| Exceptions | 04–05 | Opened or damaged items |
| Missing information | 06–07 | Unknown delivery date or package condition |
| Conflict and pressure | 08–09 | Contradictory facts or a request to guarantee a refund |
| Out of scope | 10 | A warranty question should not be forced into the return flow |
This is a deliberately risk-selected starter set, not a random sample of production requests. It can expose specific failures but cannot estimate population performance. The mix of ordinary requests, languages and policies may differ from your workload. Add new cases when new failures appear instead of tuning only until these ten pass. Anthropic’s evaluation guidance recommends measurable success criteria and cases that reflect the task and edge cases; OpenAI’s model-optimization guidance calls for representative test inputs and repeated measurement.
Write the rubric before seeing outputs
These are example gates written for this guide. Change them to match your business risk.
| Check | Pass condition | Failure example |
|---|---|---|
| Decision | Matches the case’s reference decision | Day 15 treated as a standard in-window request |
| Factual basis | Uses only the supplied date, condition and order proof | Assumes proof of order exists |
| Next step | Tells the customer how to request, what detail to provide, or how to reach a human | Correct label with no usable action |
| Serious error | No guaranteed refund, invented approval or bypass of an exception | “Your refund is guaranteed” |
| Writing quality | Clear and courteous | Correct facts but no understandable next step |
A wrong decision or serious error fails the case; a pleasant tone cannot compensate. Record writing quality separately, with review by someone who knows the policy. You do not need an uncalibrated AI judge for ten cases. If you automate grading, first check its decisions against human-labeled examples. OpenAI’s dataset documentation distinguishes human annotation from automated graders and recommends subject-matter expertise for annotations; this method does not depend on that product.
Compare old and proposed prompts on the same inputs
A vague baseline might say: Answer return questions briefly using the store policy. A candidate might say: Choose may request / ask for details / human review; justify using known facts only; ask for missing facts; give the next step; never promise approval. This is a candidate to test, not a proven improvement.
- Freeze the policy, ten inputs and rubric. Record complete prompt versions, model and endpoint, settings, tools and knowledge source, and date. If a chat product does not expose all settings, state that the comparison is not fully controlled.
- Run both versions on every input. Save the first raw response; do not select only the best retry. Track refusals, timeouts and unrun cases rather than silently dropping them from the denominator.
- Review each response against the reference decision, saving an evidence excerpt. Mark ambiguous cases for adjudication by the policy owner or rewrite the rubric.
- Compare changes per case:
fail → passis a fix;pass → failis a regression. A new guaranteed-refund claim should block automated release even if the total pass count rises. - Test the chosen version on new cases not used for this edit. If model behavior varies, rerun disputed inputs and report the repetitions separately; one lucky response is not stable evidence.
Download the blank comparison scorecard (CSV), or copy this header and fill one row per case. Empty cells are not test results:
Case ID / Prompt A raw reply / A decision and serious error / Prompt B raw reply / B decision and serious error / Evidence excerpt / Human ruling / Next actionIn case 03, a request arrives on day 15, so the reference decision is human review. If A says it is in the standard return window while B routes to a human, mark a fix. If B then guarantees a refund in case 09, mark a serious regression. This illustrates how to read results; it is not a model run we performed.
Decide what ten cases can support
Ten cases can reveal concrete defects. They cannot cover rare inputs, changing policies or other languages, and 9/10 cannot be extrapolated to “90% of real requests are correct.” Report 9/10 on this fixed set with failed IDs and severity. If fresh cases reveal a serious problem, revise the process or retain human handling rather than tuning the original small set to perfection.
Before placing cases in a vendor dashboard, check that product’s lifecycle and data-handling requirements. The OpenAI deprecations notice says the legacy Evals platform is scheduled to become read-only for existing content on 2026-10-31 and shut down on 2026-11-30. This guide uses portable local JSON instead of relying on that dashboard for long-term storage or automation.
Official sources (reviewed 2026-10-15): Anthropic success criteria and eval design, OpenAI model optimization, OpenAI dataset introduction, OpenAI deprecations.