Merchant ticket triage (BG/EN), guidelines v2

not worth it GPU worker offline

Triage incoming customer-support tickets of an online shop.

Not worth fine-tuning: baseline already meets target. The base model already reaches the target.

Pipeline

✓ Eval set generated and frozen
✓ Baseline evaluation (base model)

Baseline vs iterations

Model	Accuracy	Macro-F1	JSON validity	Exact match
Baseline (base model)	93.5%	93.5%	100.0%	69.8%

Target: macro-f1 at least 92%. Stopped: baseline already meets target.

Own tickets

Score the base model and the adapter on your own tickets. Upload tickets

Training loss

No training has finished yet.

Leakage guard and frozen eval set

Eval items: 215 (200 synthetic + 15 seed)
Frozen at: 2026-10-14 10:26:50 UTC
Blind label check: 429 of 661 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
Eval SHA-256: 78f3529dc3037330e68de05cca9ffccca2c59273391bde5f484fcb13ea1952b0

Per-class results, baseline

Class	Support	Base P	Base R	Base F1
orders	31	97%	94%	95%
shipping	31	100%	94%	97%
returns	31	100%	97%	98%
payments	31	89%	77%	83%
product_questions	30	81%	100%	90%
complaints	30	100%	100%	100%
other	31	91%	94%	92%

Confusion matrix (Baseline, rows = expected, columns = predicted)

	orders	shipping	returns	payments	product_qu...	complaints	other
orders	29	0	0	0	2	0	0
shipping	0	29	0	0	2	0	0
returns	0	0	30	1	0	0	0
payments	1	0	0	24	3	0	3
product_questions	0	0	0	0	30	0	0
complaints	0	0	0	0	0	30	0
other	0	0	0	2	0	0	29
__invalid__	0	0	0	0	0	0	0

Decisions log

No decision yet. The first one comes after the first adapter is evaluated.

Dataset preview, 10 eval items and 0 training items

Eval (frozen, used for all numbers above)

Tracking number 1Z999AA10123456784 shows 'Out for Delivery' since 8 AM but it is now 6 PM and I am still home. My package is a perishable food item and it will spoil if it isn't delivered by 8 PM. Please contact the courier immediately to ensure it arrives today.

{"category":"shipping","urgency":"high","needs_human":false} synthetic
hey, i just placed order #44556 but i forgot to add the gift wrapping option. can you still add it to the order before it goes out? i dont want to reorder everything.

{"category":"orders","urgency":"normal","needs_human":false} synthetic
имам въпрос относно доставката. избрах експресна доставка, но в потвърждението пише стандартна. дали ще дойде по-бързо или не? поръчка 4590.

{"category":"shipping","urgency":"normal","needs_human":false} synthetic
Здравейте, мога ли да получа фактура за поръчка #4492 от миналата година? Трябва ми за отчитане в фирмата. Благодаря предварително.

{"category":"payments","urgency":"normal","needs_human":false} synthetic
hey, just wondering if you guys accept paypal for the next order? also, does the 10% off code work on sale items or just full price stuff? thanks!

{"category":"payments","urgency":"low","needs_human":false} synthetic
I need to return the laptop stand I bought last month. It doesn't fit my desk. What are the steps to initiate a return? Do I need to pack it in the original box or can I just drop it off at the courier?

{"category":"returns","urgency":"normal","needs_human":false} synthetic
URGENT: My order #9921 was supposed to arrive yesterday for a critical business presentation today. The tracking shows 'out for delivery' since 8 AM but I haven't seen the courier. I am at risk of losing a major client if this doesn't arrive by 10 AM. Please contact the courier immediately.

{"category":"shipping","urgency":"high","needs_human":false} synthetic
Трети път пиша и никой не ми отговаря! Ако до утре не се реши проблемът, подавам сигнал в КЗП и повече не пазарувам от вас.

{"category":"complaints","urgency":"high","needs_human":true} seed
I returned the broken blender 10 days ago and tracking shows you received it. My refund of $45.00 still hasn't hit my card. I've already tried calling support twice with no luck. If I don't see this money by end of day, I am going to dispute this with my bank and file a chargeback.

{"category":"returns","urgency":"high","needs_human":true} synthetic
Здравейте, не става дума за конкретна поръчка, но искам да отбележа колко лошо обслужване получих от операторите на чата вчера. Никой не отговаряше и после просто затвориха сесията. Много непрофесионално поведение, очаквам обяснение от ръководството.

{"category":"complaints","urgency":"normal","needs_human":true} synthetic

Training (synthetic)

Not generated yet.

Adapters

No adapter trained yet.