Skip to content
Product
PlatformPricingDimensionsIntegrationsSecurityCompare
Solutions · by use case
Customer, guest and fan experienceProduct feedback and prioritizationRevenue, churn and retentionAgenciesContact center performanceAI agent and chatbot analyticsMulti-location performanceReviews and competitive benchmarking
Solutions · by industry
Hotels and hospitalityRestaurants and food serviceFitness and leisureRetail and e-commerceSports and live entertainmentInsuranceBanking and financial servicesSoftware and SaaSAutomotiveConsumer products and CPGContact centers and BPOsPublic sector and nonprofitReal estateGaming and betting
Learn
Artifacts GalleryCustomer storiesResearchBlogThought leadershipLessons Log in Start now
Blog

How to Classify Customer Reviews With an LLM in Python

A runnable script that sorts reviews into a fixed list of themes with structured outputs, checks every answer, caches results and grades them, run on 2,000 public complaints.

To classify customer reviews with an LLM in Python, write a closed list of themes with a one-line definition each, send every review to the model with a JSON schema whose enum fields only allow those themes, check every answer (theme on the list, evidence copied word for word from the review), cache the answers so a rerun never pays twice, and grade the labels against a labelled sample before you count anything. The script below does all of that, and every step runs on a laptop. If you would rather not maintain a script, and want reviews, tickets, surveys and calls classified with the same themes and turned into reports other teams use, Dimension Labs does this as a product, with Dimensions built for your business.

The code here was run, end to end, on 2,000 owner complaints about compact SUVs from the U.S. National Highway Traffic Safety Administration's public complaints database, because they are public data and each complaint comes with the component codes filed with it, which gives a reference to grade against. To keep this reproducible without a paid API, the published run uses a local keyword rule in place of the model; the model call is shown in full and runs with one setting. The numbers below are from that keyword rule, which makes them the floor a model run should beat.

What does the data need to look like?

One row per review, a stable id, and the text. Keep any column you might grade against or split by (a star rating, a product, a date) next to the text, and do not ask the model for anything that is already a column. The script reads JSON lines:

import json
rows = [json.loads(line) for line in open("data/complaints_2000.jsonl")]
# each row: {"id": "11552345", "text": "...", "components": "ENGINE", ...}

Product reviews from a CSV export work the same way; how to analyze a CSV of customer feedback covers cleaning the file first (empty rows, placeholder text, personal data).

How do you write the list of themes?

Read a few hundred reviews first and write down what each one is about. Group those notes into themes a team could act on, give each theme a definition in plain words (choosing what to score covers how to phrase them), and end the list with an "other" option so the model is never forced into a wrong box. For the complaints, the list has 12 themes:

THEMES = {
    "engine": "Engine stalls, loses power, burns or leaks oil, overheats, coolant loss, check-engine light.",
    "transmission": "Transmission, CVT, gear shifting, hesitation or jerking between gears, drivetrain.",
    "driver_assistance": "Automatic emergency braking, phantom braking, lane keeping, adaptive cruise "
                         "control, blind-spot or collision warnings.",
    "brakes": "Brakes, braking distance, brake pedal, ABS, parking brake.",
    # ... eight more, then:
    "other": "None of the above, or not enough detail to tell.",
}

If you do not know the themes yet, discover them first on a sample and then fix the list; topic modeling vs LLM classification compares the two approaches on the same reviews, and how AI builds feedback categories automatically shows how to turn a draft list into a fixed one.

How do you make the model return valid labels?

Use structured outputs, not a request for JSON in the prompt. The schema goes in the request, and string enums keep every answer on your list. Compare the answers to the list without regard to case: the docs note that capitalization of enum values is not guaranteed. The evidence field is the most useful one you can add: it lets you check any label against the reviewer's own words in seconds.

SCHEMA = {
    "type": "object",
    "properties": {
        "primary_theme": {"type": "string", "enum": list(THEMES)},
        "other_themes": {"type": "array", "items": {"type": "string", "enum": list(THEMES)}},
        "evidence": {"type": "string",
                     "description": "Exact words copied from the complaint that support primary_theme."},
    },
    "required": ["primary_theme", "other_themes", "evidence"],
    "additionalProperties": False,
}

def llm_classify(text, client, model):
    response = client.messages.create(
        model=model,
        max_tokens=300,
        messages=[{"role": "user", "content": build_prompt(text)}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    reply = next(b.text for b in response.content if b.type == "text")
    return json.loads(reply)

The model id comes from an environment variable, so the same script runs with whichever model you choose. One review per request keeps each answer independent of its neighbours; putting many reviews in one prompt is cheaper on instructions but lets one review's wording leak into the next one's label.

How do you run it without an API key?

Swap the model for a stub that returns the same shape. Here the stub is a keyword rule, one regular expression per theme, and the evidence is the matched words, so everything after it (validation, caching, grading, counts) runs unchanged. An excerpt:

RULES = {
    "driver_assistance": r"\b(emergency brak\w*|phantom|pre-?collision|lane (keep\w*|assist|departure)|adaptive cruise|...)\b",
    "airbags_belts": r"\b(air ?bags?|seat ?belts?)\b",
    "brakes": r"\b(brakes?|braking|abs)\b",
    # ... one pattern per theme
}

def stub_classify(text):
    t = text.lower()
    hits = [(theme, re.search(p, t)) for theme, p in RULES.items()]
    hits = [(theme, m) for theme, m in hits if m]
    if not hits:
        return {"primary_theme": "other", "other_themes": [], "evidence": ""}
    theme, m = hits[0]
    return {"primary_theme": theme, "other_themes": [h[0] for h in hits[1:]],
            "evidence": text[m.start():m.end()]}

The full script, with every pattern, runs in a few seconds with python3 classify_reviews.py. Set USE_LLM=1 and MODEL to send the same rows to a model instead.

How do you run 2,000 reviews without paying twice?

Validate every answer before you store it, write each good answer to a cache file as it arrives, and keep a retry list for the rest. A crash halfway through then costs nothing on the rerun.

def validate(label, text):
    problems = []
    if label.get("primary_theme") not in THEMES:
        problems.append("primary_theme not in list")
    if any(t not in THEMES for t in label.get("other_themes", [])):
        problems.append("other_themes not in list")
    ev = label.get("evidence", "")
    if label.get("primary_theme") != "other" and (not ev or ev not in text):
        problems.append("evidence is not an exact quote")
    return problems

def run(rows, cache_path, use_llm):
    cache = load_cache(cache_path)              # {id: label} from earlier runs
    retry = []
    with open(cache_path, "a") as out:
        for row in rows:
            if row["id"] in cache:
                continue                         # never pay twice for the same row
            label = llm_classify(row["text"], client, model) if use_llm else stub_classify(row["text"])
            problems = validate(label, row["text"])
            if problems:
                retry.append((row["id"], problems)); continue
            cache[row["id"]] = {"id": row["id"], **label}
            out.write(json.dumps(cache[row["id"]]) + "\n")
    return cache, retry

For a full file, the Message Batches API takes all 2,000 requests at once, is charged at 50% of standard prices, and most batches finish within an hour; a batch that has not finished within 24 hours expires. Each request carries the review id as its custom_id, so results map straight back to rows:

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

batch = client.messages.batches.create(requests=[
    Request(custom_id=row["id"], params=MessageCreateParamsNonStreaming(
        model=model, max_tokens=300,
        messages=[{"role": "user", "content": build_prompt(row["text"])}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}}))
    for row in rows])
# poll client.messages.batches.retrieve(batch.id) until processing_status == "ended", then:
for result in client.messages.batches.results(batch.id):
    if result.result.type == "succeeded":
        msg = result.result.message
        label = json.loads(next(b.text for b in msg.content if b.type == "text"))
        # validate(label, text_by_id[result.custom_id]) and write to the cache as above

To estimate cost before you send anything, count the tokens of a few prompts with the free token-counting endpoint and multiply by the price of the model you picked. In this run the 12 theme definitions are repeated in every prompt, and complaint text is only 30% of the input characters, so a shorter theme list is the first place to save.

How do you know the labels are right?

Grade them against a reference before you report a single count. The best reference is a sample labelled by two people from the same written definitions. The complaints come with a reference built in: the component codes filed with each one, mapped to the 11 themes other than "other".

def evaluate(rows, labels):
    stats = defaultdict(Counter)
    for row in rows:
        ref = reference_themes(row["components"])   # e.g. "SERVICE BRAKES" -> {"brakes"}
        if not ref:
            continue                                  # "UNKNOWN OR OTHER": nothing to grade
        lab = labels[row["id"]]
        pred = {lab["primary_theme"], *lab["other_themes"]} - {"other"}
        for th in THEMES:
            stats[th]["tp"] += th in pred and th in ref
            stats[th]["fp"] += th in pred and th not in ref
            stats[th]["fn"] += th not in pred and th in ref
    return stats

The run in four numbers

2,000 complaints in, one theme each, graded against the codes filed with them

2,000owner complaints classifiedseeded sample of 12,289
12themes, including otherclosed list, one definition each
1,815complaints gradablefiled with a specific component
74%primary theme matched a filed component1,350 of 1,815, keyword stub
classify_reviews.py and fetch_nhtsa.py, run 2026-11-16.

The keyword rule's primary theme matched a filed component for 1,350 of the 1,815 complaints that could be graded, or 74%. By theme, it is strong where the vocabulary is fixed (airbags, steering) and weak where people describe what happened instead of naming the part:

Keyword stub vs filed components

The keyword rule finds most engine and airbag complaints, and misses most driver-assistance ones

Precision (how often a theme the rule assigned matches a component filed with the complaint) and recall (how many complaints filed under that component the rule found), by theme. 1,815 complaints with at least one specific component.

PrecisionRecall

Engine460 complaints filed under it
66%86%
Driver assistance286 complaints filed under it
75%41%
Fuel284 complaints filed under it
84%74%
Electrical276 complaints filed under it
64%54%
Body and structure208 complaints filed under it
52%87%
Transmission196 complaints filed under it
67%44%
Brakes187 complaints filed under it
51%83%
Visibility and lights187 complaints filed under it
81%79%
Steering132 complaints filed under it
78%89%
Airbags and belts107 complaints filed under it
82%91%
Wheels, tires, suspension49 complaints filed under it
20%69%
Run on 2026-11-16: classify_reviews.py with the keyword stub, on 2,000 owner complaints from NHTSA's public complaints API. Reference: the component codes filed with each complaint.

Reading the errors tells you what to put in the theme definitions before you spend anything on a model run:

Where the keyword rule fails

The largest error groups come from words with two meanings and from owners describing events instead of parts

The largest error groups in the stub run, counted by script, and what each one means for a model run.

Error groupWhat the text showsWhat it means for your run
Wheels, tires, suspension: assigned to 133 complaints filed under other components60 of the 133 contain "steering wheel"A model reads "steering wheel" as steering; a keyword list needs an exception for every phrase like it
Brakes: assigned to 151 complaints filed under other components98 of the 151 were filed under a driver-assistance componentThe car braking by itself and the driver's brakes failing use the same words; the theme definitions have to separate them
Driver assistance: 170 of 286 complaints missedOwners describe what the car did (braking hard on its own, stopping for no reason) more often than they name the systemA model can label these from the meaning of the sentence, with no fixed keyword
Body and structure: assigned to 164 complaints filed under other components20 of the 164 mention a seat belt"Seat" inside "seat belt"; order and word boundaries in rules break quietly
The reference itself427 of 2,000 complaints list more than one component; 185 list only "unknown or other"Filed codes are a useful check; hand-label a sample before you trust any score
Counts from failure_counts.py, run 2026-11-16 on the stub labels and NHTSA component codes.

The reference has limits too. 427 of the 2,000 complaints list more than one component, and the "vehicle speed control" code covers cruise control and unintended acceleration alike. That is why a filed code, a star rating or any field you did not label yourself is a check, and a hand-labelled sample is the test. How to tell if your AI tagging is accurate covers sample sizes, precision and recall per label, and agreement between two people.

What does the output look like?

A label file with one line per review (id, primary theme, other themes, evidence) and a count per theme, ready for a pivot table or a chart:

Output of the script

Engine problems are the largest primary theme in 2,000 compact SUV complaints

Primary theme per complaint from the keyword stub. These counts are what the script writes to theme_counts.csv; a model run replaces them.

Engine
365
Body and structure
244
Fuel
224
Brakes
218
Visibility and lights
171
Other
162
Driver assistance
161
Steering
118
Airbags and belts
116
Electrical
100
Transmission
94
Wheels, tires, suspension
27
0100200300
Run on 2026-11-16: classify_reviews.py (keyword stub) on 2,000 NHTSA owner complaints about 2019 to 2022 compact SUVs. Counts are from the stub, not a language model.

Counts like these show which themes come up most. To find which themes move a star rating, a refund or a cancellation, join the labels to that outcome; how to join customer feedback to revenue data walks through the join.

Should you keep the script or use Dimension Labs?

Keep the script for a one-off question on one file. Move to a product when the same themes have to run every week, on more than one channel, with someone else reading the results. Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use. Plans start at $299 a month with no demo.

  • Fully omni-channel: reviews, tickets, chats, calls, emails and survey answers labelled with the same themes, so a theme means the same thing in every source.

  • Easy integration: connect review sites, help desks and survey tools, and join the labels to orders, accounts and locations instead of writing the join yourself.

  • Effortless enrichment: the themes in this post become Dimensions, a fixed category with an enum list and a verbatim quote beside it, which you design and test on a sample with the agent before the full run. Custom Dimensions start on Discover at $999 a month; Explore at $299 uses the pre-built library, and the join to revenue data with confidence intervals is on Prove.

  • A visual analytics layer: the agent builds the report, deck or dashboard from the labels, saves it to the Final Reports folder, exports it as PDF or PPTX, and turns the analysis into a plan you can rerun.

Stake.com replaced manual tagging this way: before, results varied across analysts and time periods; now more than 1.5 million customer conversations a month are structured with the same definitions, so one week compares with the next.

FAQ

What is the best way to classify reviews into themes with an LLM?

For a team that needs reviews classified alongside tickets, surveys and calls, with the same themes every month and reports built from them, Dimension Labs. For a one-off on a single file, the script in this post: a closed theme list, a JSON schema with enums, an exact-quote check, a cache and a graded sample.

Should I put several reviews in one API call?

It saves tokens on the instructions, but answers can bleed between reviews and one bad row can spoil the batch. One review per request, sent through the Batches API at half the standard price, is simpler to validate and retry.

Can I let the model invent the themes?

For a first look, yes: ask it to propose themes from a sample. Then fix the list. Themes that change from run to run cannot be counted or compared month to month; fixed categories or AI-written labels covers when to use each.

How many reviews do I need to label by hand?

Enough to see each theme several times. Draw a random sample, add extra rows for rare themes, and have two people label it from the same definitions before you compare the model to them.

Is a keyword rule good enough?

For themes with fixed vocabulary it can be. In this run it found 91% of airbag and seat belt complaints but 41% of driver-assistance ones, where owners describe what the car did rather than naming the system.

If you have a file of reviews and want the themes, the checks and the report without the script, start with one file.

Dimension Labs/Start with one file From $299 a month

Stop reporting. Start understanding.

Start with one file. Ask why a number moved. Plans from $299 a month, no demo and no contract on monthly plans.