FM-10 Reviewed · v1.0
Classify support tickets with a rubric and a human sample
Written category definitions, an explicit unsure option routed to people, and a weekly audit of a random sample.
You end with
Tickets labelled with a category or unsure, a routing rule for unsure, and a weekly agreement rate between model and reviewer.
- Level
- Practitioner
- Effort
- 1 h 20 min first time
- Works in
- Model API and scripts, full support
- No-code automation, full support
- Spreadsheet, partial support
- Steps
- 4 with checkpoints
Is it the right skill?
Use it when
- Ticket volume is high and categories are stable.
- A team already triages by hand and can audit a sample.
Skip it when
- Categories decide refunds, account closures or anything with legal effect.
- Your categories change every few weeks.
Before you start
- A list of categories with owners.
- Fifty or more past tickets labelled by people.
Steps
Do not move on until the checkpoint holds.
-
Write the rubric
For each category, write a definition, two examples and one near miss that belongs elsewhere.
Checkpoint Two team members label twenty past tickets with the rubric and mostly agree.
-
Allow unsure
Instruct the model to answer unsure when no definition clearly fits, and route unsure tickets to the existing manual queue.
Checkpoint Ambiguous test tickets come back as unsure, not forced into a category.
-
Test on labelled history
Run the rubric on past tickets and compare with the human labels.
Checkpoint Agreement is measured per category, not only overall.
-
Audit weekly
Each week a reviewer labels a random five percent of classified tickets without seeing the model's label.
Checkpoint The agreement rate is recorded and compared with last week.
What can go wrong
- Effect on people Misrouted tickets mean slower help. Watch the unsure rate and the per-category agreement.
- Privacy Tickets carry customer data. Check where the classification provider stores inputs.
Revision history
| Version | Date | Change |
|---|---|---|
| v1.0 | First published. |