Monitoring an LLM-powered application requires genuinely different signals than traditional application monitoring — response time and error rate alone miss the specific failure modes unique to AI-generated output.
What Traditional Monitoring Misses
An LLM call can return a completely valid HTTP 200 response, with genuinely normal response time, while still producing a fundamentally wrong, unhelpful, or subtly harmful answer — none of which shows up in conventional infrastructure monitoring dashboards at all, since from a purely technical standpoint, the request “succeeded.”
What Genuine LLM-Specific Observability Adds
log_llm_call(
prompt=prompt,
response=response,
latency_ms=elapsed,
token_count={"input": input_tokens, "output": output_tokens},
cost_usd=calculate_cost(input_tokens, output_tokens),
# LLM-specific quality signals - the genuinely important additions
user_feedback=None, # thumbs up/down, collected after the fact
contains_refusal=detect_refusal(response),
hallucination_risk_score=None, # from a separate evaluation pass, if run
grounding_source_used=retrieved_docs, # for RAG - what was ACTUALLY cited
)
Genuinely Important Metrics Beyond Latency and Errors
- Token usage and cost per request — an LLM’s real “resource consumption” isn’t CPU/memory the way traditional apps are, it’s genuinely tokens and their associated dollar cost
- User feedback signals — explicit thumbs up/down, or implicit signals like a user immediately rephrasing their question (a strong sign the first answer genuinely missed the mark)
- Groundedness — for RAG systems specifically, whether the generated answer actually, genuinely matches the retrieved source documents it was supposedly based on