📊 Enterprise LLMOps & Evaluation
Tracing & Observability in Production
Production LLM Observability with LangSmith & Langfuse, Cost & Latency Budgeting, and Real-Time User Feedback Loops
Traditional Application Performance Monitoring (APM) tools (Datadog, New Relic) monitor CPU, memory, and HTTP status codes, but they are blind to LLM failure modes. They cannot tell you if an agent got stuck in an infinite tool call loop, why Time-to-First-Token (TTFT) spiked to 3 seconds, or how much a single conversation cost in API credits. Production AI engineering requires specialized observability platforms (LangSmith, Langfuse, OpenInference) to trace multi-step execution trees and monitor real-time user feedback.
Key Concepts
Multi-Step Execution Tracing
Visualizing complex LangGraph state loops, tool calls, and sub-agent delegates in a nested trace tree to pinpoint exactly where latency spikes or failures occur.
Time-to-First-Token (TTFT) & Streaming Metrics
Tracking TTFT (initial model response latency) vs total generation time, ensuring the user interface receives the first token in <500ms even for long generations.
Token Cost Attribution & Budget Alerts
Attributing input/output token usage per user, tenant, and feature, while triggering automated kill-switches if an agent loop exceeds budget thresholds.
User Feedback Loop Integration
Binding explicit UI user ratings (thumbs up/down) directly to specific trace IDs, curating negative traces into datasets for prompt optimization.
LLM Execution Tracing & Observability Architecture
1. Langfuse SDK Setup & Tracing (TypeScript)
Wrapping LLM calls with Langfuse SDK to track generation latency, prompt versions, input/output tokens, and exact cost per invocation.
lib/tracing.tstypescript
1import Langfuse from 'langfuse';
2import { generateText } from 'ai';
3import { openai } from '@ai-sdk/openai';
4
5const langfuse = new Langfuse({
6 publicKey: process.env.LANGFUSE_PUBLIC_KEY,
7 secretKey: process.env.LANGFUSE_SECRET_KEY,
8 baseUrl: 'https://cloud.langfuse.com',
9});
10
11export async function generateWithTracing(userPrompt: string, userId: string) {
12 // 1. Create a parent trace for the request
13 const trace = langfuse.trace({
14 name: 'rag-agent-response',
15 userId: userId,
16 metadata: { env: 'production' }
17 });
18
19 // 2. Create a span for the LLM generation step
20 const generation = trace.generation({
21 name: 'gpt-4o-generation',
22 model: 'gpt-4o',
23 input: userPrompt,
24 });
25
26 const startTime = Date.now();
27 const result = await generateText({
28 model: openai('gpt-4o'),
29 prompt: userPrompt,
30 });
31
32 // 3. End generation and record metrics
33 generation.end({
34 output: result.text,
35 usage: {
36 promptTokens: result.usage.promptTokens,
37 completionTokens: result.usage.completionTokens,
38 },
39 });
40
41 return { response: result.text, traceId: trace.id };
42}2. User Feedback API Route Integration (TypeScript)
Capturing user feedback in the frontend and attaching scores (1 for thumbs up, 0 for thumbs down) directly to the specific trace ID in Langfuse.
app/api/feedback/route.tstypescript
1import { NextResponse } from 'next/server';
2import Langfuse from 'langfuse';
3
4const langfuse = new Langfuse();
5
6export async function POST(req: Request) {
7 const { traceId, score, comment } = await req.json();
8
9 // Attach explicit user feedback to the corresponding trace ID
10 await langfuse.score({
11 traceId: traceId,
12 name: 'user-explicit-feedback',
13 value: score, // 1 = Thumbs Up, 0 = Thumbs Down
14 comment: comment,
15 });
16
17 return NextResponse.json({ success: true });
18}💡
Senior Architect Insight: Logs tell you THAT an application crashed; Traces tell you WHY an LLM failed. Always bind user feedback (thumbs down) directly to the trace ID. This turns user complaints into an automated dataset pipeline for future fine-tuning.