Skip to content
Product
PlatformPricingDimensionsIntegrationsSecurityCompare
Solutions · by use case
Customer, guest and fan experienceProduct feedback and prioritizationRevenue, churn and retentionAgenciesContact center performanceAI agent and chatbot analyticsMulti-location performanceReviews and competitive benchmarking
Solutions · by industry
Hotels and hospitalityRestaurants and food serviceFitness and leisureRetail and e-commerceSports and live entertainmentInsuranceBanking and financial servicesSoftware and SaaSAutomotiveConsumer products and CPGContact centers and BPOsPublic sector and nonprofitReal estateGaming and betting
Learn
Artifacts GalleryCustomer storiesResearchBlogThought leadershipLessons Log in Start now
Blog

How to Analyze What Users Ask Your AI Agent From Trace Logs

Export the traces, keep the user's turns by session, label every conversation with an intent and an outcome, and rank intents by what follows. Worked on synthetic sample data.

To analyze what users ask your AI agent from Langfuse, LangSmith or your own trace logs, export the traces, keep only the user's messages and the agent's final replies grouped by session, label every conversation with a fixed list of intents and an outcome, and then rank the intents by what failing on them costs, for example a support ticket a few days later. Both Langfuse and LangSmith document ways to classify traces inside their own tools, which suits the engineers who run the agent. When the product, support and CX teams need those answers next to the tickets and chats that follow, with reports they can read themselves, Dimension Labs takes the exported conversations and does the labelling, the join and the reporting.

Below: where the user's question sits in a trace, how to export from each tool, how to turn traces into one row per message, how to label them, a worked example on synthetic sample data, and when the observability tool on its own is enough. For the broader difference between the two kinds of tool, see LLM observability vs conversation analytics; this post is the practical export and analysis.

Where does the user's question live in a trace?

A trace records one request to your application: the user input, the model output and every step in between. Langfuse groups the traces of one multi-turn chat into a session through a sessionId attribute, and its own page on chatbot intent analytics says each trace carries the user input, the model output and every intermediate step, with sessions grouping a conversation. For analysis you want the first and last of those (what the user asked, what they got back) and none of the retrieval chunks or tool payloads, which are large and say little about intent.

Read in each tool's docs, November 15, 2026

Each tracing tool can hand you the conversations, by a different route

Where the user's message sits, how a conversation is grouped, and the export routes each tool documents.

LangfuseLangSmithYour own OpenTelemetry spans
UnitTrace, made of observationsTrace, made of runsSpan
ConversationSession (sessionId)ThreadWhatever conversation ID your instrumentation sets
Small exportTable export from the UI as CSV or JSON, with the table's filters appliedruns.query in the SDK, or POST /api/v2/runs/queryYour collector's or backend's own export
Scheduled or bulk exportBlob storage (S3, GCS, Azure) as Parquet, CSV, JSON or JSONL, every 20 minutes to weeklyBulk export to an S3-compatible bucket as ParquetRoute spans to your warehouse
Built-in intent analysisCategorical intent scores from a batch job or an LLM-as-a-judge evaluator; dashboards on scoresInsights reports: auto-generated or predefined categories over a sample of tracesNone by default; it is your data
Sources: Langfuse Sessions, Export from UI, Export to blob storage, Chatbot analytics; LangSmith Query traces using the SDK, Bulk export trace data, Insights.

If your agent sends OpenTelemetry spans, where the user message lands depends on the instrumentation library you use, so export one conversation first and find the fields by eye before you write the full job. Langfuse says it accepts OpenTelemetry traces as well as its own SDKs, so the Langfuse routes above apply to those spans too.

How do you export traces from Langfuse or LangSmith?

Langfuse. For a one-off pull, filter the tracing table to your agent and export it as CSV or JSON from the UI. For a script, use the public API; Langfuse's docs, read on November 15, 2026, say the legacy (Langfuse v3) public API endpoints on Langfuse Cloud are removed on November 16, 2026, and that GET /api/public/traces and /observations move to GET /api/public/v2/observations, so write new jobs against the v2 route. For a standing feed, the blob storage export writes files to your bucket on a schedule; it is available on Enterprise and self-hosted, and on Pro with the Teams add-on.

LangSmith. For smaller pulls, runs.query replaces the older list_runs, which LangSmith Cloud removes on January 31, 2027. For large volumes LangSmith recommends bulk export, which writes Parquet to an S3-compatible bucket and has a 72-hour runtime limit; for customers who signed up after August 3, 2026, it is on the Enterprise plan.

Whichever route you use, export a date range, not everything, and keep the trace ID on every row. It is how an engineer gets from a failed conversation back to the trace that explains it.

How do you turn traces into one row per message?

Analysis tools read conversations, not spans. Flatten each session into its messages: the user's input on each turn and the agent's final reply, in time order, with system prompts, retrieval results and tool calls left out. Dimension Labs' conversation format needs five columns per message and keeps any other column as metadata you can filter and join on.

Field mapping

Five columns turn an exported trace into a conversation an analyst can read

From a trace export to Dimension Labs' conversation format. Field names in your export will differ by tool and SDK.

From the traceTo the columnNotes
The user's message on each turntext, with incoming = trueOnly the user's words, not the system prompt
The agent's final reply on each turntext, with incoming = falseNot the intermediate tool or retrieval steps
Session or thread IDsessionIdOne per conversation; no colons, no personal data
User IDuserIdThe end user's ID on every row, the agent's replies included
Start timedimensionlabs_timestampUnix time in milliseconds, 13 digits
Trace ID, agent version, modelExtra columnsKept as metadata, so a failure can be traced back
Columns from Dimension Labs docs: Conversation data format and Sessions and conversations.

Two checks before you label anything. Make sure a human handoff in the same chat keeps the same session ID, so the agent's turns and the person's turns read as one conversation. And strip anything a user typed that identifies them (emails, phone numbers, order numbers) before the file leaves your systems.

How do you label what users ask?

Start from a fixed list. Langfuse's intent analytics page says a predefined taxonomy keeps charts stable week over week, and suggests a clustering pass first when you do not yet know the intents. LangSmith Insights can generate categories from a sample and then save them back to the configuration so scheduled runs reuse them. Either way the end state is the same: a closed list with definitions and an "other" value.

Intent alone tells you what people ask. Add outcome fields so it also tells you where the agent lets them down. Dimension Labs' docs give a starting set: whether every request was met, some of them or none, the type of escalation, how the agent failed (did not understand, wrong answer, looping, could not do it), the unmet need in a few words, and the customer's words just before a handoff. How to find the questions your chatbot can't answer works through sorting those unmet needs into fixable gaps.

What does the analysis show?

The example below is synthetic sample data, made up for this post. A fictional project-management app runs an in-app AI assistant. One month of traces gives 6,000 sessions; each is labelled with the user's intent and whether the need was met, and joined to the support desk by user ID to see who opened a ticket within seven days. Overall, the need went unmet in 26.7% of sessions, and 81.9% of the tickets that followed came after an unmet session.

Synthetic sample data

The rarest question in the traces is the one most often followed by a support ticket

Seven intents in 6,000 sessions of a fictional in-app AI assistant, ranked two ways. Left: share of sessions. Right: share of those sessions followed by a support ticket within seven days. The "other" group is left out.

By share of sessionsBy tickets within 7 daysHow-to question31.9% of sessionsIntegration setup14.0% of sessionsBug or error report12.8% of sessionsBilling and plan11.1% of sessionsData import or export8.6% of sessionsPermissions and access8.0% of sessionsCancel or downgrade2.1% of sessionsCancel or downgrade41.1% opened a ticketIntegration setup22.0% opened a ticketBug or error report20.7% opened a ticketBilling and plan20.4% opened a ticketData import or export15.1% opened a ticketPermissions and access9.2% opened a ticketHow-to question3.9% opened a ticketBy share ofsessionsBy ticketswithin 7 daysHow-to question31.9% ofsessionsIntegrationsetup14.0% ofsessionsBug or errorreport12.8% ofsessionsBilling andplan11.1% ofsessionsData import orexport8.6% ofsessionsPermissions andaccess8.0% ofsessionsCancel ordowngrade2.1% ofsessionsCancel ordowngrade41.1% opened aticketIntegrationsetup22.0% opened aticketBug or errorreport20.7% opened aticketBilling andplan20.4% opened aticketData import orexport15.1% opened aticketPermissions andaccess9.2% opened aticketHow-to question3.9% opened aticket
Synthetic sample data made up for this post, not customer data. Script and results ship with the post.

How-to questions are the biggest group, 31.9% of sessions, and the agent handles them well: 3.9% were followed by a ticket. Users trying to cancel or downgrade are 129 sessions, 2.1% of the month, but the need went unmet in 71.3% of them and 41.1% were followed by a ticket. A dashboard of the most common intents would never show that group, and it is the one closest to lost revenue.

Should you label every conversation or a sample?

For watching the overall mix, a sample works. Langfuse's intent analytics page says a 10 to 25 percent sample is typically representative for monitoring the intent distribution and out-of-scope rate, and to classify everything when you need follow-up on each conversation. That advice holds for big intents. The questions that cost the most are often small ones, and a sample leaves them with very few sessions.

Synthetic sample data

A 10% sample is fine for the biggest intent and too thin for the rarest

Share of sessions where the need went unmet, with 95% intervals (Wilson), when every session is labelled and when a random 10% is.

How-to question: all sessions1,916 sessions; interval 9.5 to 12.3
10.8%
How-to question: 10% sample187 sessions; interval 7.5 to 16.6
11.2%
Billing and plan: all sessions667 sessions; interval 30.7 to 37.9
34.2%
Billing and plan: 10% sample77 sessions; interval 24.2 to 44.9
33.8%
Cancel or downgrade: all sessions129 sessions; interval 63.0 to 78.4
71.3%
Cancel or downgrade: 10% sample11 sessions; interval 52.3 to 94.9
81.8%
0%25%50%75%100%
Synthetic sample data made up for this post. Intervals computed in sample_calc.py.

With every session labelled, the cancel and downgrade group has 129 sessions and an unmet rate between 63.0% and 78.4%. In the 10% sample it has 11, and the interval runs from 52.3% to 94.9%, too wide to compare it with anything or to trend it month to month. It also leaves 90% of those users' conversations unread, so nobody can follow up on them. Labelling every conversation, which is how Dimension Labs enriches data, is what makes the small, costly intents visible and lets each one be acted on.

When is Langfuse or LangSmith on its own enough?

  • When the readers are the engineers who build the agent, and the question is which intents need a prompt, retrieval or tool fix. Intent scores in Langfuse or an Insights report in LangSmith sit next to the traces, cost and latency they need.

  • When the agent is internal, or the outcome you care about is visible inside the trace (an answer accepted, a thumbs up).

  • When a monthly sample of the intent mix is the whole requirement.

A second layer pays off when the outcome lives outside the trace: the ticket three days later, the cancellation, the refund. It also pays off when the readers are support, product and CX leads who want a report rather than a tracing UI, or when the same questions have to be asked of several agents and channels at once. Which chatbot and AI agent analytics tools are worth using compares the options and their prices.

What does Dimension Labs add once the traces are exported?

Dimension Labs is an omni-channel analytics agent for customer data. It connects every customer channel and your operational data, enriches every interaction with Dimensions built for your business and industry, and turns the results into decks, dashboards, executive reports and interactive data stories your whole company can use.

  • Every channel after the agent. The exported conversations sit next to the tickets, chats, calls and emails that follow, with one set of definitions, so "billing dispute" means the same thing in the assistant and in the help desk.

  • Integration with what happened next. Traces come in by file or the universal API, and on Prove ($2,999 a month) and up they join to CRM, billing and usage data, so each intent carries a cost.

  • Enrichment of every conversation. Intent, outcome, failure type and unmet need on all of them, not a sample, written in your own terms; custom Dimensions start on Discover ($999 a month).

  • Reports the whole team reads. The agent builds the monthly AI agent report, a dashboard of intents and outcomes, or a deck, as PDF or PPTX, shared by a revocable link and rerun on a schedule into the Final Reports folder.

An MCP connection, so the same Dimensions can be queried from Claude and ChatGPT, is coming soon.

Intuit QuickBooks runs its QuickBooks Assistant across more than 50 bots and over one million sessions a month. With every interaction enriched for intent, fallback drivers and escalation triggers, it reported a 35.3% reduction in its fallback (not handled) intent rate and a 57% reduction in human escalations. Read the Intuit QuickBooks story, or see AI agent and chatbot analytics.

FAQ

How do I see what users ask my chatbot in Langfuse?

Classify each conversation into intents and write the labels back as categorical scores, from a batch job over the API or with an LLM-as-a-judge evaluator, then chart them in a custom dashboard. That is the method Langfuse documents. To read the same intents next to tickets and cancellations, export the sessions and analyze them where that data lives.

Can I analyze LangSmith traces outside LangSmith?

Yes. Query runs with runs.query for smaller pulls, or use bulk export to Parquet for large volumes, then flatten each thread into messages with a session ID, user ID and timestamp.

What is the best tool to analyze what users ask an AI agent?

For intents and outcomes on every conversation, joined to the tickets and revenue that follow and reported to support, product and CX, Dimension Labs. For engineers fixing prompts and retrieval inside the agent, the intent features in Langfuse or LangSmith Insights.

How many intents should the list have?

Enough that "other" stays small, and few enough that each intent has an owner. Split an intent when its unmet needs point to different fixes, and merge two when they always lead to the same one.

To report the result to leadership each month, what to report to executives about your AI agent sets out a seven-line report.

To try it on your own agent, export a week of sessions with the user's messages, the agent's replies, a session ID and a timestamp, and start with one file of conversations. Every plan starts with a 14-day free trial, with no demo or sales call.

Dimension Labs/Start with one file From $299 a month

Stop reporting. Start understanding.

Start with one file. Ask why a number moved. Plans from $299 a month, no demo and no contract on monthly plans.