FM-04 Reviewed · v1.1
Build a small evaluation set before you change a prompt
Twenty to fifty saved cases with expected properties, run before and after every change, so improvements are measured, not felt.
You end with
A versioned file of test cases, a baseline score for the current prompt, and a per-case comparison for each change.
- Level
- Practitioner
- Effort
- 1 h 45 min first time
- Works in
- Model API and scripts, full support
- Coding agent in a repository, full support
- Spreadsheet, partial support
- Chat assistant, partial support
- Steps
- 5 with checkpoints
Is it the right skill?
Use it when
- A prompt runs in a product or a recurring workflow.
- Several people propose prompt changes and you need to decide.
Skip it when
- The prompt is for a one-off task; review the output directly instead.
Before you start
- Access to real-shaped inputs that you may store after removing personal data.
- A place to keep cases next to the prompt, under version control if possible.
Steps
Do not move on until the checkpoint holds.
-
Collect cases
Gather twenty to fifty inputs: typical ones, known failures, and edge cases such as empty input or very long input. Remove personal data.
Checkpoint At least a quarter of the cases are difficult ones.
-
Write expected properties
For each case, write what must be present, what must be absent and any format rule. Avoid exact expected text; wording legitimately varies.
Checkpoint Two people would grade the same output the same way.
-
Record a baseline
Run the current prompt on every case and grade it. Save outputs and grades with the prompt version.
Checkpoint You have a single baseline number and per-case results.
-
Change one thing and re-run
Make one change, re-run all cases and compare case by case, not only the total.
Checkpoint You can list which cases got better and which got worse.
-
Keep the set alive
Whenever a new failure appears in use, add it as a case before fixing it.
Checkpoint The case count grows over time and each case has a source note.
What can go wrong
- Privacy Saved cases outlive the conversation they came from. Remove personal data before saving.
- Accuracy Graders drift towards the version they prefer. Grade without knowing which version produced the output.
Revision history
| Version | Date | Change |
|---|---|---|
| v1.1 | Added the rule to add new failures as cases before fixing them. | |
| v1.0 | First published. |