Store review triage (BG) v2 + own tickets
completedSort customer reviews of a Bulgarian online store by what the shop should do with them: thank the customer, fix a problem, look at both, or consider an idea. Public, read-only report built from the stored results.
Baseline vs iterations
| Model | Accuracy | Macro-F1 | JSON validity | Exact match |
|---|---|---|---|---|
| Baseline (base model) | 90.2% | 90.3% | 100.0% | 90.2% |
| Iteration 1 served | 97.7% | 97.7% | 100.0% | 97.7% |
Target: macro-f1 at least 95%. Stopped: target met (0.977 >= 0.950).
Own tickets
533 of the shop's own tickets, masked and frozen as a separate slice. They came with labels made by people: the dataset authors Georgieva-Trifonova, Stefanova and Kalchev (Harvard Dataverse, doi:10.7910/DVN/TXIK9P, CC BY 4.0): the 182 test reviews of the MTEB release plus the 351 reviews of the check half of its train split. Neither the judge nor Chirak changed them. The tickets themselves are private to the run owner.
Training included 348 own tickets of run Store review triage (BG) v2.
| Model | Accuracy | Macro-F1 | category |
|---|---|---|---|
| Base model | 73.7% | 61.4% | 73.7% |
| Served adapter (iteration 1) | 82.6% | 69.9% | 82.6% |
Accuracy is on the primary field; the field columns are the accuracy of each field on its own.
Per-class results, baseline vs iteration 1
| Class | Support | Base P | Base R | Base F1 | P | R | F1 | F1 vs base |
|---|---|---|---|---|---|---|---|---|
| compliment | 54 | 100% | 91% | 95% | 100% | 100% | 100% | +4.8 pts |
| complaint | 54 | 88% | 93% | 90% | 96% | 98% | 97% | +7.2 pts |
| mixed | 54 | 98% | 80% | 88% | 98% | 94% | 96% | +8.5 pts |
| suggestion | 53 | 80% | 98% | 88% | 96% | 98% | 97% | +9.1 pts |
Confusion matrix (Iteration 1, rows = expected, columns = predicted)
| compliment | complaint | mixed | suggestion | __invalid_... | |
|---|---|---|---|---|---|
| compliment | 54 | 0 | 0 | 0 | 0 |
| complaint | 0 | 53 | 1 | 0 | 0 |
| mixed | 0 | 1 | 51 | 2 | 0 |
| suggestion | 0 | 1 | 0 | 52 | 0 |
| __invalid__ | 0 | 0 | 0 | 0 | 0 |
Leakage guard and frozen eval set
- Eval items
- 215 (200 synthetic + 15 seed)
- Frozen at
- 2026-10-15 10:34:04 UTC
- Blind label check
- 15 of 276 eval candidates dropped because the generator, labelling the text blind with the student's prompt, disagreed with the target label. The kept items are the unambiguous ones, so this eval is cleaner and easier than an unfiltered one.
- Eval SHA-256
- 77abc3cabb1c3ec342873a92cff95bbcd83b05371cccc58ee4f11e531913372d
| Train set | Candidates | Dropped as near-copies of eval | Drop rate | Dropped by label check |
|---|---|---|---|---|
| Iteration 1 | 1367 | 19 | 1.4% | 73 |
Decisions log
The model was not consulted: the stop rules ended the loop.