Own tickets: Store review triage (BG)
Upload 100 to 1000 of your own tickets. Emails, phone numbers, IBANs, card numbers and long digit runs are masked on this server before anything is stored or sent to Claude. Claude labels each ticket twice; you review the labels, freeze the set, and the base model and the served adapter are scored on it.
Status
- Tickets
- 0
- Labelled
- 0
- Unstable
- 0
- Not judged yet
- 0
- Dropped
- 0
Judge usage: 0 requests, 0 input and 0 output tokens.
Own tickets
182 of the shop's own tickets, masked and frozen as a separate slice. They came with labels made by people: the dataset authors Georgieva-Trifonova, Stefanova and Kalchev (Harvard Dataverse, doi:10.7910/DVN/TXIK9P, CC BY 4.0), test split of the MTEB release. Neither the judge nor Chirak changed them. The tickets themselves are private to the run owner.
79% of the uploaded tickets were set aside for training by the split given in the upload: 702 went into a training file, 182 were kept for evaluation. 2 training tickets were left out because they repeated an evaluation ticket or shared a word 8-gram with one. The two parts never overlap.
A follow-up run trained on the training part: Store review triage (BG) + own tickets.
| Model | Accuracy | Macro-F1 | category |
|---|---|---|---|
| Base model | 75.8% | 62.3% | 75.8% |
| Served adapter (iteration 1) | 83.5% | 69.8% | 83.5% |
Accuracy is on the primary field; the field columns are the accuracy of each field on its own.