Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

This post introduces a framework for monitoring generative AI applications beyond traditional golden signals. It details specific metrics for cost (token creep, model drift, uncached calls, tagging), safety (prompt injection, PII leakage, toxicity, jailbreaks), and quality (hallucination rate, relevance, user satisfaction, completeness, RAG retrieval quality). The talk highlights how LLM applications break traditional monitoring assumptions and proposes actionable metrics and strategies for effective observability.