How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems | Datadog

What this post added

This post details the optimization of LLM usage for writing postmortems, focusing on cost, quality, and safety. It describes the integration of structured metadata from Datadog's Incident Management app with unstructured Slack discussions, fed into an ensemble of LLM models to generate postmortem drafts. Key challenges addressed include data quality and hallucinations, cost/speed/quality trade-offs, and trust/privacy. Solutions involved refining LLM instructions with both structured and unstructured data, establishing an experimentation framework with a custom API for data extraction and structuring, and implementing safeguarding mechanisms like prioritizing information, lowering model temperature, and adjusting LLM parameters to reduce hallucinations and improve reliability. The post also mentions an adjacent project for LLM-generated incident summaries that informed this work.