How to Build a Codeframe for Open-Ended Survey Responses
A codeframe is the list of codes, definitions and rules used to tag open-ended survey answers consistently. Here are the five steps to build one, an example codeframe and agreement test, and how codeframes relate to Dime…
A codeframe is the list of codes you use to tag answers to an open-ended survey question, with a definition and rules for each code so that two people tagging the same answer choose the same codes. You build one by reading a sample of responses, drafting codes from what people actually wrote, writing rules for what each code includes and excludes, testing whether two coders agree, and revising until they do.
This post walks through those five steps with an illustrative codeframe and agreement test, then shows how the same idea scales up to Dimensions, which apply a codeframe to every record automatically.
What a codeframe contains
Each code needs five things: a short name, a one-sentence definition, what it includes, what it excludes (usually pointing to the code that takes it instead), and one or two example answers. Codes are often grouped into nets, such as "Product" or "Commercial", so results can be reported at two levels. Most codeframes also carry "Other" and "No reason given" so that every answer lands somewhere.
Step 1: Read a sample
Start from the responses, not from the codes you expect. Pew Research Center, which codes large open-ended studies, builds its codebooks by iteratively reviewing random samples of responses and mapping the concepts that recur. Draw from every score band, because detractors and promoters talk about different things, and keep reading until new batches stop producing new themes.
Step 2: Draft codes from what people wrote
Name codes in your customers' terms and at the level someone can act on. "Product issues" is too broad to give to a team; "setup and onboarding" and "missing feature" are not. Decide early whether an answer can take more than one code. It usually should: in Pew's study, codes were applied to each unit of information in a response, and while a median of 73% of respondents mentioned only one topic, the rest mentioned several.
Step 3: Write the rules
Most disagreement between coders comes from answers that sit between two codes. Rules settle them in advance. Two kinds do most of the work:
Boundary rules that say which code wins when two seem to fit. The codeframe behind our Chicago café analysis separates order accuracy from food quality with an example pair: "'They forgot my soup' = order accuracy; 'the soup was watery' = food quality."
Priority rules for a "main reason" field, when only one code may be chosen. The same café codeframe ranks problems in a fixed order and tells the coder to output the highest one, not the last one narrated, so "the dining room was filthy and then they overcharged me" is coded as a billing error, which ranks above cleanliness.
An example codeframe (illustrative)
This codeframe is made up for a fictional software product's NPS follow-up question, "What is the main reason for your score?". The example answers are invented.
Illustrative codeframe
Each code says what it includes, and where the near-misses go instead
A made-up codeframe for a fictional software product's NPS follow-up, "What is the main reason for your score?". The example answers are invented.
| Code | Definition and rules |
|---|---|
| Setup and onboarding Net: Product | Getting started took too long or was confusing Includes: Account setup, data import, first-week training Excludes: Day-to-day confusion after setup (Ease of use) Example: "Took us a month to get our data in." |
| Missing feature Net: Product | Something the customer needs is not in the product Includes: Named feature requests, integrations that do not exist Excludes: A feature that exists but breaks (Bugs and reliability) Example: "No way to export to our warehouse." |
| Bugs and reliability Net: Product | The product fails, errors or is slow Includes: Outages, errors, sync failures, slow pages Excludes: Requests for new features (Missing feature) Example: "Reports time out every Monday." |
| Ease of use Net: Product | Using the product day to day is hard Includes: Navigation, finding settings, too many clicks Excludes: Initial setup (Setup and onboarding) Example: "I can never find the filters." |
| Price and plan Net: Commercial | Cost, value or plan limits Includes: Price level, renewal increase, seat limits Excludes: Billing mistakes (Support experience) Example: "Too expensive for a team our size." |
| Support experience Net: Service | Contact with support or account staff Includes: Response time, quality of answers, billing fixes Excludes: The bug itself (Bugs and reliability) Example: "Support took three days to reply." |
| Other Net: None | A clear reason that fits no code Includes: Anything specific not covered above Excludes: Blank or non-answers (No reason given) Example: "Your logo changed." |
| No reason given Net: None | No usable reason Includes: Blank, "n/a", "fine", a score repeated Excludes: Short but specific answers Example: "9" |
Each row is a rule a second coder can follow without asking you. The Excludes line does the most work, because it says where the near-misses go.
Step 4: Test agreement
Have two people code the same sample independently, then compare code by code. Raw percent agreement flatters rare codes, because two coders who both leave a code off most answers agree by default. Cohen's kappa corrects for the agreement expected by chance; 1 is perfect and 0 is no better than chance. A common bar is 0.7: Pew treats a Krippendorff's alpha, a related chance-corrected measure, of 0.7 or higher as acceptable, and reached 0.70 overall across 25 codes in its democracy study.
The test below is illustrative. Two coders tag the same 80 answers using the codeframe above. Five codes clear the bar. "Ease of use" agrees on 83.8% of answers but has a kappa of 0.54, because it overlaps with setup and onboarding and the coders split the overlap differently.
Illustrative data
One code fails the agreement test despite 84% raw agreement
Cohen's kappa for each code when two coders tag the same 80 answers, with raw agreement under each label. Made-up counts for this post.
Dashed line: the 0.7 bar
Step 5: Revise and freeze
Fix the codes that fail, not the coders. For a code like "Ease of use" that means sharpening the boundary ("anything in the first 30 days goes to Setup and onboarding"), splitting a code that holds two ideas, or merging two codes coders cannot tell apart. Then recode a fresh sample and test again. Pew tested its draft codebook over five rounds on samples of at least 75 responses before handing it to a wider team. Once it passes, give the codeframe a version number and change it only on purpose, so this quarter's results compare with last quarter's.
How a codeframe relates to Dimensions
A Dimension in Dimension Labs is a codeframe for one question about each record, applied by a language model to every response, review or ticket rather than a sample. Each value has a written definition, and the rules from step 3, boundary and priority alike, are part of the Dimension's description. Our Chicago café analysis read 1,995 public reviews against 27 Dimensions, which is 53,865 structured readings.
From the café brief
One review, 27 small codeframes: the Dimensions read from every café review
Each Dimension has its own values and rules. 27 Dimensions on 1,995 reviews gave 53,865 structured readings.
| Group | Dimensions | Count |
|---|---|---|
| Sentiment and subject | Overall sentiment, visit type, primary topic, primary friction | 4 |
| The people | Staff interaction, staff named, staff role, praise driver | 4 |
| The system axes | Order accuracy, digital channel, speed and wait, billing at the register | 4 |
| The craft axes | Bread and bakery, food quality, price and value, cleanliness | 4 |
| Loyalty and outcome | Loyalty signal, recommendation, issue resolution, competitor mention | 4 |
| Plus seven more | Competitor context, issue severity, the operational owner of each fix, the pain point and the delight in the customer's own words, an exact-quote verbatim, a one-line factual summary of the visit | 7 |
| Total | 27 |
Two things carry over from manual coding. Many Dimensions replace one long codeframe: instead of a single list where "wrong order at the drive-thru" must pick a code, there is one field for the problem and another for the channel. And the agreement test still applies, with the model as one coder and a person as the other. Our post on how to check AI tagging accuracy covers that test, and survey text analysis covers what to do with the coded answers. A codeframe is a small data taxonomy, and applying one by model is a form of automated data labeling.
FAQ
How many codes should a codeframe have?
Enough that "Other" stays small and every code is used, and few enough that coders can hold the rules in their head. For one survey question that is often somewhere between eight and twenty, grouped into a handful of nets.
What is the difference between a codeframe and a codebook?
The terms are mostly interchangeable. Market research tends to say codeframe; academic and public-opinion research tends to say codebook, and a codebook may also hold instructions for the coders.
Can AI build the codeframe for me?
A model can draft candidate codes from a sample quickly, which saves the first reading pass. The decisions about boundaries, priorities and what level a team can act on still need a person who knows the business, and the result still needs an agreement test.
Dimension Labs builds business-specific Dimensions from your surveys, reviews and tickets, applies them to every record, and joins the results to your customer and revenue data so each code can be tied to what it costs. See customer intelligence, or start with one file of survey responses.