Merchant ticket triage (BG/EN), guidelines v2 + real tickets (ABCD train)
completedTriage incoming customer-support tickets of an online shop. Public, read-only report built from the stored results.
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 91.6% | 91.7% | 100.0% | 65.1% |
| Iteration 1 | 96.3% | 96.2% | 100.0% | 90.7% |
| Iteration 2 served | 97.7% | 97.7% | 100.0% | 92.6% |
Target: exact-match at least 92%. Stopped: target met (0.926 >= 0.920).
Own tickets
146 of the shop's own tickets, masked and frozen as a separate slice. Labels: 128 by the Claude judge, 0 by a person and 18 by Claude Code on the owner's delegation (not a person) (4 dropped in review). The tickets themselves are private to the run owner.
| Model | Accuracy | Macro-F1 | category | urgency | needs_human |
|---|---|---|---|---|---|
| Base model | 67.8% | 58.5% | 67.8% | 47.9% | 76.7% |
| Served adapter (iteration 2) | 71.2% | 60.2% | 71.2% | 80.1% | 85.6% |
Accuracy is on the primary field; the field columns are the accuracy of each field on its own.
Per-class results, baseline vs iteration 2
| Class | Support | Base P | Base R | Base F1 | P | R | F1 | F1 vs base |
|---|---|---|---|---|---|---|---|---|
| orders | 31 | 100% | 87% | 93% | 97% | 94% | 95% | +2.0 pts |
| shipping | 31 | 96% | 87% | 92% | 97% | 100% | 98% | +6.9 pts |
| returns | 31 | 100% | 100% | 100% | 100% | 100% | 100% | +0.0 pts |
| payments | 31 | 93% | 84% | 88% | 100% | 97% | 98% | +10.2 pts |
| product_questions | 30 | 76% | 97% | 85% | 97% | 100% | 98% | +13.1 pts |
| complaints | 30 | 93% | 93% | 93% | 97% | 97% | 97% | +3.3 pts |
| other | 31 | 88% | 94% | 91% | 97% | 97% | 97% | +6.2 pts |
Confusion matrix (Iteration 2, rows = expected, columns = predicted)
| orders | shipping | returns | payments | product_qu... | complaints | other | __invalid_... | |
|---|---|---|---|---|---|---|---|---|
| orders | 29 | 1 | 0 | 0 | 1 | 0 | 0 | 0 |
| shipping | 0 | 31 | 0 | 0 | 0 | 0 | 0 | 0 |
| returns | 0 | 0 | 31 | 0 | 0 | 0 | 0 | 0 |
| payments | 1 | 0 | 0 | 30 | 0 | 0 | 0 | 0 |
| product_questions | 0 | 0 | 0 | 0 | 30 | 0 | 0 | 0 |
| complaints | 0 | 0 | 0 | 0 | 0 | 29 | 1 | 0 |
| other | 0 | 0 | 0 | 0 | 0 | 1 | 30 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-14 11:06:16 UTC
- Blind label check
- 338 of 549 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- 9c785baa706da25c68a789d16f25796b09563660e15545ca453efb524fb0c08e
| Train set | Candidates | Dropped as near-copies of eval | Drop rate | Dropped by label check |
|---|---|---|---|---|
| Iteration 1 | 3176 | 129 | 4.1% | 1957 |
| Iteration 2 | 3266 | 144 | 4.4% | 2028 |
Decisions log
claude-sonnet-5-5: Exact match is 0.907 against a 0.92 target. Category accuracy is high at 0.963 and the remaining errors are boundary confusions. Orders is the weakest class (recall 0.871), mostly confused with payments, shipping, returns and product_questions. Complaints versus other is the second confusable pair, with low-stakes feedback and legal or privacy threats landing on the wrong side. Targeted boundary data should close the gap. (model's own stop opinion: continue, not used)
- orders_vs_payments_shipping x250 Tickets about modifying or cancelling an order that has not shipped. Include changing the payment method, adding or forgetting a promo code, cancelling right after payment, and changing the delivery address before dispatch. The category is orders when the customer wants the order itself changed or cancelled. Contrast with real payments tickets (wrong or double charge, invoice) and real shipping tickets (parcel already dispatched). Mix Bulgarian and English, calm and angry tones, and include ALL-CAPS messages.
- orders_vs_returns_wrong_quantity x150 Wrong contents or quantity in an order, such as fewer or different items than ordered, appearing in the confirmation or in what was shipped. Contrast with returns tickets where the customer wants to send back a faulty or damaged item. Include deadlines and threats to leave a bad review, which set urgency to high. Mix Bulgarian and English.
- orders_vs_product_questions_pre_checkout x150 Customers with items in the cart or an unpaid or unplaced order who ask to change the color, size or quantity. Contrast with pure pre-purchase questions about specs, stock or compatibility. Urgency is low in both. Make clear when the request is to edit an order or cart and when it is only a question about a product. Mix Bulgarian and English.
- complaints_low_severity_feedback x200 Mild, low-urgency feedback with no concrete order to fix, such as excessive packaging, slow replies, an unhelpful or curt agent, or a dented box where the item is fine. These belong to complaints, not other, and are low urgency with needs_human false when the tone is mild. Contrast with praise, which is other, and with real returns tickets for damaged items. Mix Bulgarian and English.
- other_privacy_legal_vs_complaints x150 Data, privacy and account requests that include threats of legal action or of reporting to a regulator. Examples include marketing emails after account deletion, GDPR access or erasure requests, and account compromise. The category is other, urgency is high, and needs_human is true. Contrast with complaints about rude staff or non-responsive support that threaten to report the shop but have no data or account issue. Mix Bulgarian and English.
- payments_vs_orders_shipping_regression x100 Balanced regression examples for payments, shipping and returns, kept clearly distinct from orders. Include charges, refunds not tied to a return, invoices, delivery address or date questions for parcels already dispatched, and refunds for returned items. Vary language, tone and formatting so the model does not over-predict orders.
The model was not consulted: the stop rules ended the loop.