Building production AI for regulated industries with a leading digital insurer

This post details the engineering effort to build a production-ready AI system for a regulated insurance industry, focusing on compliance, accuracy, and speed. Key technical contributions include: a state-of-the-art knowledge retrieval system with dynamic chunking, hybrid retrieval (dense + sparse), and multi-stage reranking; a novel evaluation harness constructor that systematically discovers failure modes through spectrum sampling, meta-error analysis, hierarchical clustering, and evaluation synthesis; a meta-evaluation process to refine evaluation prompts and align with expert judgment; genetic optimization to generate gold training examples; supervised fine-tuning of Qwen3-32B on these gold examples, achieving performance competitive with frontier models; and reinforcement learning (GRPO) for tool use, specifically optimizing retrieval query formulation and handling ambiguous questions. The architectural decision to move beyond traditional vector embeddings by incorporating these advanced techniques is also highlighted.