The 25-Minute AI Output Acceptance Test

You are helping me build an acceptance test for one recurring AI-generated deliverable.

Deliverable: [EXAMPLE: CUSTOMER-REPLY DRAFT, RESEARCH MEMO, SALES-CALL SUMMARY]
Business purpose: [WHAT DECISION OR ACTION THIS OUTPUT SUPPORTS]
Intended reader: [ROLE]
Good examples: [PASTE 2–3 REAL ACCEPTED OUTPUTS]
Bad or corrected examples: [PASTE 1–3 FAILURES OR EDITS]
Non-negotiable facts, format, tone, or policy requirements: [LIST]

Extract a grading rubric with:
- 4–6 dimensions that matter to the business purpose
- a precise PASS condition for each dimension
- observable failure signals
- automatic-fail conditions
- evidence from the examples that supports each rule

Rules:
- Do not use vague labels such as “high quality,” “professional,” or “accurate” without defining observable evidence.
- Keep factual correctness separate from style.
- Do not infer a requirement from one example unless you mark it as a proposed rule for human confirmation.
- End with the smallest set of questions needed to finalize the rubric.

Build a compact acceptance-test set for this recurring AI workflow.

Workflow: [DESCRIBE THE INPUT → OUTPUT JOB]
Rubric: [PASTE THE CONFIRMED RUBRIC]
Typical inputs: [PASTE 3–5 ANONYMIZED EXAMPLES]
Known failures: [PASTE EXAMPLES OR DESCRIBE THEM]

Create 10 test cases:
- 5 representative everyday cases
- 3 difficult edge cases
- 2 cases that should trigger an automatic fail or human escalation

For each case provide:
- case ID
- input or input specification
- why it belongs in the set
- rubric dimensions it tests
- expected properties of a passing output
- failure to watch for

Do not write ideal answers that leak into the generation prompt. Keep expected results as observable properties. Remove or replace private information before returning the set.

Grade AI outputs against the confirmed acceptance test.

Rubric: [PASTE]
Test cases: [PASTE]
Candidate outputs: [PASTE OUTPUTS WITH CASE IDS]

For every case:
1. Evaluate each rubric dimension independently.
2. Cite the exact output text or omission that supports the grade.
3. Return PASS, FAIL, or HUMAN REVIEW for each dimension.
4. Apply every automatic-fail rule before calculating the overall result.
5. Return one overall case result: ACCEPT, REJECT, or HUMAN REVIEW.

Then summarize:
- pass rate by rubric dimension
- repeated failure patterns
- any rubric rule that was ambiguous in practice
- the smallest prompt, context, or workflow change worth testing next

Do not reward fluent wording when the output misses facts, instructions, source support, or required structure. Do not invent evidence that is absent from the candidate output.

Turn these evaluation results into a production release and sampling policy.

Workflow: [NAME]
Rubric and automatic-fail rules: [PASTE]
Test results: [PASTE]
Business consequence of a bad output: [LOW / MEDIUM / HIGH, WITH EXPLANATION]
Current human review capacity: [EXAMPLE: 20 ITEMS PER WEEK]

Define:
- minimum test-set pass threshold before the workflow can move into production
- automatic-fail conditions that always block release
- which outputs always require human review
- a random production sample size and review cadence
- what to log for every sampled output
- triggers for pausing the workflow
- the evidence required to change the prompt, model, tools, or rubric
- a simple version label for the prompt, test set, rubric, and model

Make the policy proportionate to consequence. Do not recommend zero review for high-consequence work. A model change, prompt change, tool change, or material input-distribution change must trigger re-testing.