FM-05 Reviewed · v1.0
Compare two models or prompts blind on your own task
Side-by-side outputs with hidden labels, scored on a short rubric, so the choice reflects your work rather than a reputation.
You end with
A tally of wins, losses and ties per criterion, and a decision with the tie rule written down in advance.
- Level
- Practitioner
- Effort
- 1 h 10 min first time
- Works in
- Model API and scripts, full support
- Spreadsheet, full support
- Chat assistant, partial support
- Steps
- 4 with checkpoints
Is it the right skill?
Use it when
- You are choosing between tools, models or prompt versions for recurring work.
- Opinions in the team differ and you want evidence.
Skip it when
- Cost or a policy rule already decides the choice.
- You have fewer than about ten realistic inputs.
Before you start
- Ten or more realistic inputs.
- A colleague who can score without seeing which option produced which output.
Steps
Do not move on until the checkpoint holds.
-
Write the rubric first
Pick two to four criteria, such as correctness, completeness and tone, with a one-line description of a good answer for each.
Checkpoint The rubric and the tie rule are written before any output is seen.
-
Generate and shuffle
Run both options on every input. Randomly assign each pair to left and right, and record the key separately.
Checkpoint The reviewer's sheet shows only Left and Right.
-
Score blind
The reviewer picks left, right or tie for each criterion and each case.
Checkpoint Every cell is filled; ties are allowed.
-
Unblind and tally
Apply the key and count wins per option per criterion.
Checkpoint The decision follows the pre-written rule, including what happens on a tie.
What can go wrong
- Accuracy A small sample can mislead. Treat a narrow win on ten cases as a tie.
- Cost Running two options doubles usage for the trial period.
Revision history
| Version | Date | Change |
|---|---|---|
| v1.0 | First published. |