Store review triage (BG)
completedSort customer reviews of a Bulgarian online store by what the shop should do with them: thank the customer, fix a problem, look at both, or consider an idea. Public, read-only report built from the stored results.
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 90.2% | 90.3% | 100.0% | 90.2% |
| Iteration 1 served | 98.1% | 98.1% | 100.0% | 98.1% |
Target: macro-f1 at least 95%. Stopped: target met (0.981 >= 0.950).
Own tickets
182 of the shop's own tickets, masked and frozen as a separate slice. They came with labels made by people: the dataset authors Georgieva-Trifonova, Stefanova and Kalchev (Harvard Dataverse, doi:10.7910/DVN/TXIK9P, CC BY 4.0), test split of the MTEB release. Neither the judge nor Chirak changed them. The tickets themselves are private to the run owner.
79% of the uploaded tickets were set aside for training by the split given in the upload: 702 went into a training file, 182 were kept for evaluation. 2 training tickets were left out because they repeated an evaluation ticket or shared a word 8-gram with one. The two parts never overlap.
A follow-up run trained on the training part: Store review triage (BG) + own tickets.
| Model | Accuracy | Macro-F1 | category |
|---|---|---|---|
| Base model | 75.8% | 62.3% | 75.8% |
| Served adapter (iteration 1) | 83.5% | 69.8% | 83.5% |
Accuracy is on the primary field; the field columns are the accuracy of each field on its own.
Per-class results, baseline vs iteration 1
| Class | Support | Base P | Base R | Base F1 | P | R | F1 | F1 vs base |
|---|---|---|---|---|---|---|---|---|
| compliment | 54 | 100% | 76% | 86% | 100% | 100% | 100% | +13.7 pts |
| complaint | 54 | 98% | 98% | 98% | 98% | 98% | 98% | +0.0 pts |
| mixed | 54 | 92% | 89% | 91% | 98% | 96% | 97% | +6.6 pts |
| suggestion | 53 | 76% | 98% | 86% | 96% | 98% | 97% | +11.2 pts |
Confusion matrix (Iteration 1, rows = expected, columns = predicted)
| compliment | complaint | mixed | suggestion | __invalid_... | |
|---|---|---|---|---|---|
| compliment | 54 | 0 | 0 | 0 | 0 |
| complaint | 0 | 53 | 1 | 0 | 0 |
| mixed | 0 | 0 | 52 | 2 | 0 |
| suggestion | 0 | 1 | 0 | 52 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-15 08:00:29 UTC
- Blind label check
- 7 of 276 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- 7993e19745108ce278d9c7e248106e265fe3142e3b12acd70ac81ac634a8b342
| Train set | Candidates | Dropped as near-copies of eval | Drop rate | Dropped by label check |
|---|---|---|---|---|
| Iteration 1 | 1368 | 11 | 0.8% | 19 |
Decisions log
The model was not consulted: the stop rules ended the loop.