How Accurate Is AI at Coding Open-Ended Survey Responses?
In a 2024 study of survey answers coded into 50 categories, the best language model got 93.9% right against 94.7% for a second human. Where models miss, and the check to run on your own codes.
On clear categories, AI codes open-ended survey responses about as accurately as a second trained human coder: in a 2024 study that coded survey answers into 50 categories, the best language model was right 93.9% of the time against 94.7% for a second human. It falls further behind on vague or catch-all codes and on topics it has not seen, so accuracy depends more on the codeframe and the check you run than on the model. For an insights team that needs every verbatim coded the same way every wave, checked against human labels and joined to scores and behavior, Dimension Labs is the platform we would choose: it applies Dimensions written for your business to every response across surveys, reviews, tickets and calls, and builds the reports from those fields.
Below: what four published studies and our own research found, where the models miss, and the check that tells you whether your own coding is good enough. Our posts on how to tell if your AI tagging is accurate and whether an AI assistant gives the same answer twice cover the measurement in more depth.
How accurate is AI at coding open-ended survey responses compared to human coders?
The study closest to a survey team's job is Mellon and colleagues' 2024 paper in Research and Politics. They took answers to the British Election Study's "most important issue" question, coded into 50 categories, and compared six language models, three supervised machine learning methods and a second human coder. Two scenarios matter:
No coded data yet. The best model, Claude-1.3 with a few examples in the prompt, scored 93.9% against 94.7% for the second human, and beat a supervised model trained on 1,000 hand-coded cases (93.5%).
Coding new text after earlier waves were coded. The best model scored 80.9%, the human 88.6%, and a supervised model trained on 576,000 cases 77.8%. The gap to the human here is 7.7 points (our arithmetic: 88.6 minus 80.9).
Published study
On new survey answers the best model came within a point of a second human coder; on novel text it fell 7.7 points behind
Accuracy coding British Election Study "most important issue" answers into 50 categories. Two scenarios from the paper.
The same paper found that two of the six models, PaLM-2 and Llama-2, did substantially worse than the best models on every metric, so "AI" is not one accuracy figure. The model you use, and how you prompt it, moves the number.
Does GPT agree with human coders on survey verbatims?
Often more consistently than humans agree with each other, and that is a different thing from being right. Gilardi, Alizadeh and Kubli coded four samples of tweets and news articles, 6,183 texts in all, for relevance, stance, topics and frames. ChatGPT's accuracy against trained annotators beat crowd workers by about 25 percentage points on average, and its agreement with itself across two runs averaged 91% at the default temperature and 97% at 0.2, against 79% between two trained annotators and 56% between crowd workers.
A model that gives the same answer twice can give the same wrong answer twice. Pangakis, Wolken and Fasching replicated 27 annotation tasks from 11 published social science datasets with GPT-4. The median task reached 0.85 accuracy, but median precision was only 0.65, and 9 of the 27 tasks had precision or recall below 0.5. For a third of tasks, the model missed at least half the true cases, flagged more false ones than true ones, or both.
Published study
Accuracy looked strong across 27 coding tasks; precision is where a third of them broke
GPT-4 against human labels on 27 annotation tasks from 11 social science datasets. Dot: median task. Line: middle half of tasks (25th to 75th percentile).
Their check on consistency is useful for survey teams: labelling each answer several times and keeping only the ones the model labelled the same way every time picked out the easy cases. Accuracy was 19.4 points higher on answers with a consistency score of 1.0, and 85.1% of answers had one. The rest are the ones a person should read.
Where does AI coding go wrong on open-ended responses?
In the same places human coders disagree. Von der Heyde and colleagues coded 5,072 German survey answers about why people take part in a panel survey, against two expert coders who agreed with a kappa of 0.91. The best model, GPT-4o, reached a macro F1 of around 0.7 without fine-tuning. It did very well on clear reasons such as incentives, interest and fun (F1 around 0.9) and very poorly on the catch-all codes, "no reason" and "other" (F1 between 0.02 and 0.3), and only a fine-tuned model reached what the authors call satisfactory performance. Because the misses were uneven across codes, the share of answers in each code also came out different from the human coding.
Three practical lessons follow:
Write codes a stranger could apply. "Other" and "general dissatisfaction" are hard for people and harder for models. Our post on building a codeframe for open-ended responses covers include and exclude rules.
Expect drift when the topic mix changes. The 7.7-point gap in Mellon's second scenario came when the model coded novel text after earlier waves had been coded; recheck after a big event or a new question wording.
Judge by code, not overall. An overall accuracy near the 0.85 median in Pangakis et al. sat alongside tasks where the model found fewer than half the true cases.
Five studies, one table
Across five studies, models came close to human coders, and the gaps sat in particular tasks and codes
What each study coded, what it compared the model with, and the headline result.
| Study | Text coded | Compared with | Result |
|---|---|---|---|
| Mellon et al. 2024 | Open-ended survey answers (UK, 50 categories) | Second human coder; supervised models | 93.9% vs 94.7% for the human; 80.9% vs 88.6% on novel text |
| Gilardi et al. 2023 | 6,183 tweets and news articles; relevance, stance, topics and frames | Crowd workers; trained research assistants | About 25 points above crowd workers on average; run-to-run agreement 91% to 97%, against 79% between trained annotators |
| Pangakis et al. 2023 | 27 tasks from 11 published datasets | Human labels from the original studies | Median accuracy 0.85, median F1 0.707; 9 of 27 tasks had precision or recall below 0.5 |
| von der Heyde et al. 2025 | 5,072 German survey answers on reasons to take part | Two expert coders (kappa 0.91) | Macro F1 around 0.7 for the best model; only a fine-tuned model reached satisfactory performance |
| Dimension Labs research | About 10,000 MLB fan survey comments | The fan's own 0 to 10 rating | Within one point 67% of the time, exact 36%; 87% exact agreement across scoring runs |
What did our own research find on fan survey comments?
Our published research had GPT-4.1 read single open-ended comments from about 10,000 fan surveys across five Major League Baseball teams and predict the 0 to 10 experience rating each fan gave. Two-thirds of predictions fell within one point of the fan's own rating (67%), 36% matched exactly, and three scoring runs agreed exactly 87% of the time. The predictions ran about one point below the fans' own scores, and the paper reads that as a difference in what the two measure: the rating is the fan's overall verdict, the text records the moments that stood out. The full method is in LLM Predictive Scoring and Validation.
How do I check whether AI coding is accurate enough for my survey?
Run the same test the studies ran, on your own answers, before the coded data goes into a report:
Draw a random sample of a few hundred answers, with extra cases from rare codes, and have two people code it independently from the written codeframe.
Check that the two people agree with each other first. If they do not, fix the definitions before judging the model.
Compare the model's codes with the resolved human codes one code at a time: agreement, kappa, precision and recall, next to a baseline that always guesses the most common code.
Run the model two or three times and read the answers where its runs disagree; those are your edge cases.
Repeat after any change to the prompt, the model, the question wording or the codeframe.
If the model matches your second coder on the codes you report, use it on every answer. The real gain in all four studies is coverage: the full set coded the same way every wave, where people can only code a sample.
Which tool should an insights team trust to code verbatims at scale?
Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.
For a survey program, that means four things. Survey answers are read in the same place as reviews, tickets, chats and calls, so a theme coded in the post-visit survey means the same thing in a complaint email. Survey platforms such as Medallia and Qualtrics connect alongside your operational data, so a code can be read next to the NPS score, the visit or the renewal it belongs to. Every response is enriched with Dimensions written for your business and industry, applied to every answer every wave (custom Dimensions start on Discover at $999 a month, and the join to CRM and revenue data with confidence intervals is on Prove at $2,999). And the agent turns the coded results into decks, dashboards, PDF and executive reports and interactive data stories the rest of the company can open.
The NBA moved its fan surveys to Dimension Labs and now analyzes 100,000+ open-ended responses within hours instead of months, with insight review cycles down from about 60 days to under 6 hours. See the NBA story.
When something smaller is the better choice: for one survey a year with a few hundred answers, two people with a spreadsheet and a codeframe will do, and a general AI assistant is fine for a first read. Our comparison of tools to analyze open-ended survey responses lists research-focused options too.
FAQ
Is AI as accurate as human coders for open-ended survey responses?
Close, on well-defined codes. In Mellon et al., the best model scored 93.9% against 94.7% for a second human coder. It trailed by more on new text and on catch-all codes.
Which AI is best for coding survey verbatims?
For coding verbatims across surveys and other channels with fixed Dimensions, checked against human labels and joined to scores and outcomes, Dimension Labs. Among raw models the published results vary by model and task: in Mellon et al., two of six models did substantially worse than the best. Test any of them on a coded sample first.
How many responses should a human code to check the AI?
A few hundred, drawn at random, with extra cases from rare codes so each code has enough examples. Checking AI tagging accuracy shows how sample size sets the margin of error.
Does the AI give the same codes if I run it again?
Mostly. Gilardi et al. found 91% to 97% agreement between two runs, and our MLB study found 87% exact agreement across three runs. Our repeatability test shows why a fixed list of codes matters more than the model.
To see how close your own coding gets, start with one file of survey responses, code it against your codeframe, and compare a sample with your coders' labels. Plans start at $299 a month with no demo.