Skip to content
Product
PlatformPricingDimensionsIntegrationsSecurityCompare
Solutions · by use case
Customer, guest and fan experienceProduct feedback and prioritizationRevenue, churn and retentionAgenciesContact center performanceAI agent and chatbot analyticsMulti-location performanceReviews and competitive benchmarking
Solutions · by industry
Hotels and hospitalityRestaurants and food serviceFitness and leisureRetail and e-commerceSports and live entertainmentInsuranceBanking and financial servicesSoftware and SaaSAutomotiveConsumer products and CPGContact centers and BPOsPublic sector and nonprofitReal estateGaming and betting
Learn
Artifacts GalleryCustomer storiesResearchBlogThought leadershipLessons Log in Start now
Blog

How LLM Analytics Improve Accuracy & Performance

As large language models (LLMs) become embedded in apps across customer service, product support, and digital experiences, one question still remains unanswered: how do you know your AI is performing well?

LLM analytics is the practice of measuring how an application built on a large language model performs in production: what it costs, how fast it responds, and whether its answers were accurate, on-brand and actually solved the user's problem. It combines observability data (tokens, latency, errors, traces) with analysis of the conversations themselves, so a team can see which topics the model fails on, why, and what to change.

Most teams already have the first half. The second half is where accuracy and customer experience improve, because a model can answer every request quickly and cheaply and still leave many users without what they came for.

For which tools cover each half, and what they cost, see which chatbot and AI agent analytics tools are worth using.

Two layers of LLM analytics

Dimension Labs guide

Observability says the system is up; conversation quality says whether it helped

The two layers of LLM analytics, side by side.

ObservabilityConversation quality
MeasuresTokens and cost, latency (p95, p99), errors, tool calls, traces per requestWhether the need was met, helpfulness, user effort, groundedness, off-policy or off-brand answers, handoffs and their causes
AnswersIs the system fast, cheap and up?Did it help, and where does it fail?
Comes fromRequest logs and tracesReading each conversation, by a person or a model, against fixed definitions
Typical toolsTracing platformsEvals, transcript analysis, analytics agents
Dimension Labs, from the layers described in this guide.

For the first layer, tracing tools such as Langfuse ("open source agent evals and observability"), LangSmith (which records each action an agent takes to find issues with "latency, cost, and response quality") and Helicone ("AI Gateway & LLM Observability") cover latency, cost and traces. Several also run evaluations on traces. The second layer needs the conversations read for meaning, at full volume, with the same definitions every time.

To get from those traces to a ranked list of what users ask, how to analyze what users ask your AI agent shows the export and labelling steps.

The conversation-quality metrics that matter

  • Need met: did the user get what they asked for, partly, or not at all? Count each need, since one conversation can hold several.

  • Cause of failure: the topic is not covered, the model misunderstood, the answer was wrong or incomplete, a rule blocked it, or the user asked for a person.

  • Helpfulness and effort: how many turns, rephrasings or restarts it took.

  • Groundedness: whether the answer matches the source content, or the model made something up, such as a policy that does not exist.

  • Brand and policy: tone, promises the company cannot keep, advice it should not give.

  • Outcome after the chat: a handoff, a repeat contact, a refund or a cancellation, joined from other systems.

A worked example: why containment can mislead

The sample report below was run on synthetic data for a fictional cinema chain's assistant. Of 4,074 conversations, 83.0% stayed with the bot, a containment rate most dashboards would show in green. Read need by need, 43.5% of the 4,961 customer needs were not met. A containment rate counts every conversation that never reached a person as a success, including the ones where the customer left without an answer.

Illustrative data

Most conversations stayed with the bot, and almost half the needs in them went unmet

Headline numbers from a sample chatbot report, read one customer need at a time. Partial answers count as half a need met.

4,074conversations, holding 4,961 customer needs
46.6%would answer no to the helpfulness questionreweighted for who answers
Stayed with the botshare of conversations
83.0%Containment
Needs not metshare of customer needs
43.5%
0%25%50%75%100%
Redrawn from the summary of a Dimension Labs sample chatbot report. Illustrative data: synthetic conversations for a fictional cinema chain, September 2026.

The same report shows where the failures concentrate: three topics hold 70.6% of the unmet needs, and the report lists 7 fixes ready to build. That is the list a team can act on this sprint: new content for topics the model was never given, prompt or retrieval changes for topics it misreads, and routing rules for requests that should go to a person.

Illustrative data

Three topics hold 70.6% of the needs not met

Each topic's share of the needs the bot did not meet, largest first.

Showtimes
33%
Refunds and exchanges
24%
Membership
13%
Ticket prices
10%
Theater locations
8%
Gift cards
7%
Food and drink
5%
0%10%20%30%
Redrawn from Exhibit 2 of the same Dimension Labs sample chatbot report. Illustrative data: synthetic conversations for a fictional cinema chain.

The method for counting escalations by cause is in how to measure why customers escalate from a chatbot.

How to set it up

  1. Fix the definitions first. Write down the topics, outcomes and causes before labelling, or the counts will drift from week to week.

  2. Label every conversation. A language model can do the reading. Run it on all of them, so smaller topics show up.

  3. Check the labels. Read a random set yourself (we use about 120) without looking at the labels, then compare. Where the logs record a handoff or fallback, the labels should agree.

  4. Cut by version. Tag each conversation with the model, prompt and knowledge-base version, so a change can be compared before and after.

  5. Join to outcomes. Link conversations to repeat contacts, refunds or renewals, so fixes are ranked by what they are worth.

Product loop

Ask a question of your conversations and get a chart

The agent writes the queries over the labelled conversations and answers with a chart.

ask
Dimension Labs, recorded in the product.

Teams that measured bot conversations this way have reported large gains: QuickBooks reported 57% fewer human escalations across more than 50 bots.

For how to define a resolved conversation, see containment rate vs resolution rate vs deflection rate.

Checking intent or topic labels against human coders is covered in how to tell if your AI tagging is accurate.

To check whether an AI agent's resolutions hold up, see how to audit its resolved conversations.

Frequently Asked Questions (FAQs)

What is the difference between LLM analytics and LLM observability?

Observability tracks the system: latency, cost, errors and traces. LLM analytics includes that and adds whether the conversations worked for users, by topic and cause.

For a side-by-side of the tools on each side, and a worked sample of 2,000 sessions where most failures had a clean trace, see LLM observability vs conversation analytics.

How are LLM evals different from LLM analytics?

Evals usually test a model against a fixed set of prompts before or after a change. LLM analytics reads what real users asked in production, which finds the topics no test set covered.

What metrics are used in LLM analytics?

Token usage, cost, latency and error rates on the system side; need met, cause of failure, helpfulness, effort, groundedness, policy breaches and handoffs on the conversation side.

Can an LLM grade another LLM's conversations?

Yes, if the definitions are fixed and the grading is checked against a human-read sample. Without that check, the grader's mistakes look like the bot's.

Dimension Labs reads every conversation your AI agents have, labels each need, outcome and cause, and joins them to your business data. Dimension Labs is an AI analytics agent that reads every data source a business has: tickets, chats, calls, emails, reviews and surveys, along with CRM, billing, orders, usage and location data. It learns how the business works, extracts hundreds of business-specific Dimensions (typed fields on every record), and shows what drives revenue, retention and cost, with effects and confidence intervals on the Prove plan and up. The agent then builds the reports, decks, dashboards and alerts and keeps them running on a schedule. Plans start at $299 a month with no demo or sales call. See AI agent analytics or what an AI analytics agent is. Or start with one file.

Dimension Labs/Start with one file From $299 a month

Stop reporting. Start understanding.

Start with one file. Ask why a number moved. Plans from $299 a month, no demo and no contract on monthly plans.