How to Tell if Your AI Tagging Is Accurate
Compare the model's labels with people labelling the same records, and read percent agreement next to Cohen's kappa. A worked example, sample sizes and numbers from our research.
To tell if AI tagging is accurate, have people label a random sample of the same records independently, then compare their labels with the model's using percent agreement and a chance-corrected measure such as Cohen's kappa, one label at a time. Raw agreement alone can look excellent when a label is rare, so always read it next to kappa, precision and recall, and next to what a model that guessed the most common answer would score.
Below: what each measure tells you, a worked kappa calculation, how many records to check, and real numbers from our published validation research.
What you are comparing
A tagging check compares two readings of the same records: the model's labels and a reference. Usually the reference is a trained person working from the same written definitions (a codebook). Sometimes it is something the customer reported, such as the rating they left.
Human labels are a reference with errors of their own. In content analysis, agreement between two trained human coders on subjective tasks typically lands around a kappa of 0.60 to 0.75. In one study that compared survey respondents' own coding of their open-ended answers with trained coders' labels, kappa was 0.46. So the first job is to check that two people agree with each other. If they do not, the definition needs work before the model is judged against it.
Why percent agreement misleads
Percent agreement is the share of records where the model and the reference gave the same label. It is easy to explain and it overstates quality whenever one answer dominates. McHugh notes that it does not account for agreement by chance and may overestimate true agreement.
Our study of 70,450 support conversations shows the trap with real numbers. On the 3,911 conversations customers rated, the model's predicted satisfaction agreed with the customer's own favorable-or-unfavorable verdict 91% of the time. But 91% of ratings were favorable, so a rule that called every conversation favorable would score about the same. The paper therefore leads with skew-resistant measures: precision of 0.51 when flagging a low rating and a Matthews correlation of 0.35, against 0.29 and 0.22 for sentiment. Rebuilding the two-by-two table from those published figures gives a Cohen's kappa of about 0.34. That comparison is against the customer's rating rather than a human coder, but the lesson carries over: the kappa of 0.34 describes the tagger far better than the 91% does.
Dimension Labs research
91% agreement, and a rule that calls everything favorable scores almost the same
Agreement with the customer's own favorable-or-unfavorable verdict on 3,911 rated support conversations. Kappa for the model, rebuilt from the published figures: about 0.34.
Cohen's kappa, worked through
Kappa compares observed agreement with the agreement you would expect if both raters labelled at random with their own base rates: κ = (po − pe) / (1 − pe). A kappa of 1 is perfect agreement and 0 is chance level.
Illustrative data
Worked kappa: 92% agreement, kappa 0.74
Illustrative data, not from a real project: a person and a model each label the same 200 reviews for one field, “mentions a billing problem”. Agreeing cells in bold.
| Model: yes | Model: no | Total | |
|---|---|---|---|
| Human: yes | 30 | 10 | 40 |
| Human: no | 6 | 154 | 160 |
| Total | 36 | 164 | 200 |
| Measure | Calculation | Value |
|---|---|---|
| Observed agreement po | (30 + 154) / 200 | 0.92 |
| Chance agreement pe | (40/200 × 36/200) + (160/200 × 164/200) = 0.036 + 0.656 | 0.692 |
| Cohen's kappa κ | (0.92 − 0.692) / 0.308 | 0.74 |
| Precision | 30 / 36 | 0.83 |
| Recall | 30 / 40 | 0.75 |
| A model that always says no | agrees 160 / 200 times | 80%, κ = 0 |
How good is 0.74? The bands McHugh attributes to Cohen call 0.61 to 0.80 "substantial" and 0.81 to 1.00 "almost perfect". McHugh argues they are too lenient for health research and treats anything below 0.60 as inadequate. For business tagging, compare against the human-to-human kappa on the same label. If two people reach 0.70 and the model reaches 0.68 against each of them (hypothetical numbers), the model is labelling about as well as a second person would.
The other numbers to report
Precision and recall per label. Precision is how often a "yes" from the model is right. Recall is how many of the true "yes" records it found. A flag used to route work needs high precision; a flag used to count a problem needs both.
Within-one agreement for scores. For a 0 to 10 rating, exact matches undersell a model that is usually one point off. In our MLB study, predictions matched the fan's own rating exactly 36% of the time and fell within one point 67% of the time, across about 10,000 surveys from five teams.
Run-to-run consistency. Score the same records twice. In the MLB study, three independent runs gave the same rating 87% of the time pairwise and within one point 99.9% of the time; 80.9% of sessions got an identical score on all three runs.
Agreement after a change. Swapping the model changes labels. In our follow-up paper, GPT-4.1-mini agreed exactly with the customized GPT-4.1 setup on only 47% of sessions. Recheck after every prompt or model change.
Where it fails. Break agreement down by segment. Agreement in the MLB study was 83% within one point for fans who rated 9 or 10, and 54% for fans who rated 7 or 8, where the text mixes praise and complaint.
Dimension Labs research
The model's own confidence flag separates reliable labels from shaky ones
Share of predicted 0 to 10 experience ratings within one point of the rating the fan gave, about 10,000 MLB fan surveys from five teams.
How many records to check
Two things set the size: how precise you need the estimate to be, and how many positive examples the sample contains.
Illustrative data
Below about 200 records, the kappa range is too wide to tell a good tagger from a poor one
The same illustrative table (true kappa 0.74, 92% agreement, 20% of records positive), resampled at different sizes. Dot: true kappa. Line: 95% range of the estimate.
Dashed line: 0.60, the floor McHugh treats as adequate
| Records | Agreement ± | Positives at 5% |
|---|---|---|
| 50 | ±7.5 pts | 2.5 |
| 100 | ±5.3 pts | 5 |
| 200 | ±3.8 pts | 10 |
| 400 | ±2.7 pts | 20 |
| 800 | ±1.9 pts | 40 |
In this example, 200 double-labelled records put kappa somewhere between about 0.61 and 0.85, enough to tell a usable tagger from a poor one. Rare labels are the harder problem. If 5% of records mention a safety issue, a random sample of 200 contains about 10 of them, too few to say anything about recall. Either sample about 1,000 records for that label, or build a stratified sample that deliberately includes more of the rare cases and weight the results back. The hotel schema check below used a stratified sample for exactly this reason.
Setting up a check, step by step
Write the codebook. One definition per label, with an example of a yes, a no and a borderline case. Decide the rule for records that mention two things at once.
Draw the sample. Random, plus extra rows for rare labels, long records, other languages and very short records.
Have two people label it blind. Neither sees the model's answer or the other person's. Compute their kappa first.
Resolve disagreements between the two people. Where they differ, agree the right label and tighten the codebook.
Compare the model with the resolved labels. Per label: agreement, kappa, precision, recall, with the majority-answer baseline next to them.
Read the disagreements. Most are a vague definition, an edge case the codebook missed, or a record that genuinely says two things. Fix the definition, not just the score.
Run the automatic checks too. Before the Orlando hotel analysis, a 49-review stratified sample (fees, cleanliness, named staff, very short and very long reviews, four other languages) was checked by script: zero labels outside the allowed values, zero quotes that were not exact passages from the review. One ambiguous case led to a rule change. These checks catch broken output. They cannot tell whether a judgment was right, so they sit alongside the human comparison.
Repeat on a schedule. After any change to the prompt, the model or the kind of data coming in, and on a small fresh sample every month.
Product loop
Labels that pass the check become columns you can ask questions of
A plain-English question answered with a chart, from fields extracted from every record.
FAQ
What is a good Cohen's kappa for AI tagging?
Compare it with two people on the same label. As a rule of thumb, McHugh treats 0.60 as the floor for adequate agreement, and trained human coders on subjective tasks typically reach 0.60 to 0.75.
Should I use Cohen's kappa or Krippendorff's alpha?
Cohen's kappa compares two raters on one label. Krippendorff's alpha works with any number of raters, missing labels and ordered or numeric scales. For a model against one adjudicated reference on a yes-or-no label, kappa is enough.
Is accuracy against customer ratings the same as accuracy against human coders?
No. A rating is the customer's overall verdict, and the text often describes one moment. Our MLB study found predicted ratings ran about one point below the fans' own, and read that gap as a difference in what the two measure rather than pure error.
Dimension Labs extracts typed fields from every ticket, chat, call, email, review and survey, and our published research shows how we validate them. Once the labels hold up, they join your CRM, billing and usage data so you can see what drives retention and cost; see customer intelligence or the related posts on topic modeling vs LLM classification and automated data labeling. Plans start at $299 a month with no sales call. Start with one file.