Merchant ticket triage (BG/EN), guidelines v2, target exact match
completedTriage incoming customer-support tickets of an online shop. Public, read-only report built from the stored results.
Baseline exact-match
65.6%
Final exact-match
94.9%
Change
+29.3 pts
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 93.0% | 93.1% | 100.0% | 65.6% |
| Iteration 1 served | 100.0% | 100.0% | 100.0% | 94.9% |
Target: exact-match at least 92%. Stopped: target met (0.949 >= 0.920).
Per-class results, baseline vs iteration 1
| Class | Support | Base P | Base R | Base F1 | P | R | F1 | F1 vs base |
|---|---|---|---|---|---|---|---|---|
| orders | 31 | 100% | 94% | 97% | 100% | 100% | 100% | +3.3 pts |
| shipping | 31 | 96% | 84% | 90% | 100% | 100% | 100% | +10.3 pts |
| returns | 31 | 100% | 94% | 97% | 100% | 100% | 100% | +3.3 pts |
| payments | 31 | 90% | 90% | 90% | 100% | 100% | 100% | +9.7 pts |
| product_questions | 30 | 81% | 100% | 90% | 100% | 100% | 100% | +10.5 pts |
| complaints | 30 | 91% | 100% | 95% | 100% | 100% | 100% | +4.8 pts |
| other | 31 | 97% | 90% | 93% | 100% | 100% | 100% | +6.7 pts |
Confusion matrix (Iteration 1, rows = expected, columns = predicted)
| orders | shipping | returns | payments | product_qu... | complaints | other | __invalid_... | |
|---|---|---|---|---|---|---|---|---|
| orders | 31 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| shipping | 0 | 31 | 0 | 0 | 0 | 0 | 0 | 0 |
| returns | 0 | 0 | 31 | 0 | 0 | 0 | 0 | 0 |
| payments | 0 | 0 | 0 | 31 | 0 | 0 | 0 | 0 |
| product_questions | 0 | 0 | 0 | 0 | 30 | 0 | 0 | 0 |
| complaints | 0 | 0 | 0 | 0 | 0 | 30 | 0 | 0 |
| other | 0 | 0 | 0 | 0 | 0 | 0 | 31 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-14 10:41:45 UTC
- Blind label check
- 328 of 555 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- 77a3168e5890918e24e09483a802e36d7576a112cf7a1ccc6c3b0b776d12d53a
| Train set | Candidates | Dropped as near-copies of eval | Drop rate | Dropped by label check |
|---|---|---|---|---|
| Iteration 1 | 2984 | 106 | 3.6% | 1797 |
Decisions log
After iteration 1
Stop rules: stop, target met (0.949 >= 0.920)
The model was not consulted: the stop rules ended the loop.