7 lessons for IT leaders on using observability to monitor AI applications

This post contributes lessons learned on monitoring AI applications, specifically focusing on the need for LLM observability. Key technical takeaways include: 1. Implementing LLM observability from the MVP stage to capture usage history for ROI calculation. 2. Utilizing probabilistic metrics like token consumption, retrieval quality, and use-case classification alongside traditional metrics for AI systems. 3. Adopting OpenTelemetry semantic conventions for generative AI to standardize telemetry vocabulary and avoid rebuilds when frameworks or models change. 4. Ensuring instrumentation reports actual provider-billed figures (e.g., token counts) rather than estimations to avoid cost discrepancies. 5. Managing observability costs by avoiding high-cardinality tags (like conversation IDs) and uniform sampling, and instead computing derived values at query time. 6. Debugging AI quality issues by capturing retrieval data (what was retrieved, ranking, support for the answer) to differentiate between model, ranking, and content problems. 7. Emphasizing the creation of robust evaluation systems for AI quality assurance.