Store review triage (BG) + own tickets
completedSort customer reviews of a Bulgarian online store by what the shop should do with them: thank the customer, fix a problem, look at both, or consider an idea. Public, read-only report built from the stored results.
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 90.2% | 90.3% | 100.0% | 90.2% |
| Iteration 1 served | 97.7% | 97.7% | 100.0% | 97.7% |
Target: macro-f1 at least 95%. Stopped: target met (0.977 >= 0.950).
Own tickets
182 of the shop's own tickets, masked and frozen as a separate slice. They came with labels made by people: the dataset authors Georgieva-Trifonova, Stefanova and Kalchev (Harvard Dataverse, doi:10.7910/DVN/TXIK9P, CC BY 4.0), test split of the MTEB release. Neither the judge nor Chirak changed them. The tickets themselves are private to the run owner.
Training included 700 own tickets of run Store review triage (BG).
| Model | Accuracy | Macro-F1 | category |
|---|---|---|---|
| Base model | 75.8% | 62.3% | 75.8% |
| Served adapter (iteration 1) | 83.5% | 71.5% | 83.5% |
Accuracy is on the primary field; the field columns are the accuracy of each field on its own.
Per-class results, baseline vs iteration 1
| Class | Support | Base P | Base R | Base F1 | P | R | F1 | F1 vs base |
|---|---|---|---|---|---|---|---|---|
| compliment | 54 | 100% | 78% | 88% | 98% | 98% | 98% | +10.7 pts |
| complaint | 54 | 95% | 98% | 96% | 98% | 98% | 98% | +1.8 pts |
| mixed | 54 | 96% | 87% | 91% | 96% | 96% | 96% | +5.0 pts |
| suggestion | 53 | 76% | 98% | 86% | 98% | 98% | 98% | +12.2 pts |
Confusion matrix (Iteration 1, rows = expected, columns = predicted)
| compliment | complaint | mixed | suggestion | __invalid_... | |
|---|---|---|---|---|---|
| compliment | 53 | 0 | 1 | 0 | 0 |
| complaint | 0 | 53 | 1 | 0 | 0 |
| mixed | 1 | 0 | 52 | 1 | 0 |
| suggestion | 0 | 1 | 0 | 52 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-15 08:30:47 UTC
- Blind label check
- 6 of 276 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- b40fb79db38f286689fc772b11e98ac1c3d2645bc0ad27be463a331cb8d45198
| Train set | Candidates | Dropped as near-copies of eval | Drop rate | Dropped by label check |
|---|---|---|---|---|
| Iteration 1 | 1372 | 13 | 1.0% | 22 |
Decisions log
The model was not consulted: the stop rules ended the loop.