LLM Observability (Tracing, Logging and Monitoring)

What is it?

When an AI app misbehaves you need to see exactly what it was asked, what it retrieved, which tools it called and what it answered. Observability tools record that trail — and because the trail may contain private information, it must be handled carefully.

Also known as: LLM monitoring, tracing LLM calls, OpenTelemetry GenAI, prompt logging, LLM tracing, AI application monitoring, LangSmith, LLM analytics.

Technical definition

Practices: structured tracing of spans per step (retrieval, model call, tool call), correlation IDs, token and cost accounting, latency percentiles, error and refusal rates, sampling for human review, feedback capture, and turning production failures into regression tests (How to Evaluate AI Systems and Outputs). Standards such as the OpenTelemetry generative-AI semantic conventions are under development (verify status). Many commercial and open-source platforms exist; evaluate data residency and retention before sending traces to a third party.

Entity type: technique · Freshness class: volatile.

When should I use it?

Needed as soon as an AI feature has real users.

Prerequisites and related concepts

Understand these first: How to Evaluate AI Systems and Outputs, MLOps (Machine Learning Operations).

Related concepts: Model Drift and Production Monitoring, LLM Cost Optimization, AI Privacy and Security: What to Know Before You Paste Data, Agent Evaluation, Model Serving and Inference Infrastructure, AI Governance.

Implementation patterns

• Trace spans for retrieval, generation and tools.
• Redact PII/secrets before storage.
• Sample failed or low-rated traces for review.
• Add production failures to the eval set.
• Dashboards for cost, latency and error rates.

Technologies and frameworks

Commonly associated: OpenTelemetry, LangSmith, Langfuse, MLflow tracing, application logs. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

1. Define what to capture.
2. Redact sensitive data.
3. Instrument each step.
4. Set retention and access policies.
5. Review samples weekly and feed evals.

Common failure modes

• Logging raw sensitive prompts indefinitely.
• No trace linkage across steps.
• Cost blind spots (retries, reasoning tokens).
• Alert fatigue.
• Third-party trace storage without data review.

Troubleshooting

• Cannot reproduce a bad answer — Record full inputs, retrieved context, parameters and model version.
• Sensitive data in logs — Add redaction at ingestion and shorten retention.

Contrasted with

• Model Drift and Production Monitoring — Drift monitoring watches data/performance statistics over time; LLM observability traces individual request flows.

Questions this page answers

• What is LLM observability?
• How do I monitor an LLM application?
• What should I log from LLM calls?
• How do I track token cost?
• Is it safe to log prompts?

Typical goals: debug production LLM issues; monitor cost and quality; log safely.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

• Status of OpenTelemetry GenAI conventions.
• Platform names, features and licences.

Canonical sources (unverified leads):

• OpenTelemetry semantic conventions for generative AI — OpenTelemetry GenAI semantic conventions (now maintained in a dedicated repository linked from opentelemetry.io/docs/specs/semconv/; status: in development as of 2026-10-09)
• Provider and platform observability documentation — locator to be identified

Related concepts