Merchant ticket triage (BG/EN), guidelines v2
not worth itTriage incoming customer-support tickets of an online shop. Public, read-only report built from the stored results.
Fine-tuning was skipped: baseline already meets target.
Baseline macro-f1
93.5%
Final macro-f1
—
Change
—
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 93.5% | 93.5% | 100.0% | 69.8% |
Target: macro-f1 at least 92%. Stopped: baseline already meets target.
Per-class results, baseline
| Class | Support | Base P | Base R | Base F1 |
|---|---|---|---|---|
| orders | 31 | 97% | 94% | 95% |
| shipping | 31 | 100% | 94% | 97% |
| returns | 31 | 100% | 97% | 98% |
| payments | 31 | 89% | 77% | 83% |
| product_questions | 30 | 81% | 100% | 90% |
| complaints | 30 | 100% | 100% | 100% |
| other | 31 | 91% | 94% | 92% |
Confusion matrix (Baseline, rows = expected, columns = predicted)
| orders | shipping | returns | payments | product_qu... | complaints | other | __invalid_... | |
|---|---|---|---|---|---|---|---|---|
| orders | 29 | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| shipping | 0 | 29 | 0 | 0 | 2 | 0 | 0 | 0 |
| returns | 0 | 0 | 30 | 1 | 0 | 0 | 0 | 0 |
| payments | 1 | 0 | 0 | 24 | 3 | 0 | 3 | 0 |
| product_questions | 0 | 0 | 0 | 0 | 30 | 0 | 0 | 0 |
| complaints | 0 | 0 | 0 | 0 | 0 | 30 | 0 | 0 |
| other | 0 | 0 | 0 | 2 | 0 | 0 | 29 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-14 10:26:50 UTC
- Blind label check
- 429 of 661 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- 78f3529dc3037330e68de05cca9ffccca2c59273391bde5f484fcb13ea1952b0
Decisions log
No decision yet. The first one comes after the first adapter is evaluated.
Training loss
No training has finished yet.