How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems | Datadog

This post details the development of an LLM-based feature within Bits AI to assist in writing postmortems. It describes the challenges of using LLMs in production, including data quality, hallucinations, cost/speed/quality trade-offs, and trust/privacy. The solution involves combining structured metadata from Datadog's Incident Management app with unstructured Slack discussions, feeding this into an ensemble of LLM models. Key technical efforts include establishing an experimentation framework for data extraction and LLM instruction refinement, using techniques like lowering model temperature and adjusting LLM parameters to reduce hallucinations and improve reliability. The post also mentions the development of LLM-generated incident summaries as a testing ground for rapid iterations on incident data.