Enhancing Rovo Chat with Hybrid LLM Approach

This post details the technical implementation of Rovo Chat, an AI-powered assistant, focusing on its hybrid LLM approach. Key contributions include: 1. Query Rewriting: Two-stage process (semantic and keyword queries) using smaller LLMs like LLaMA-3 8B, achieving +10% answer precision and -17% latency. 2. Accurate Information Retrieval: Multi-plugin routing using a BERT-based intent classifier and parallel searches, with LLM plugin accuracy ranging from 73.1% to 94.2%. 3. Plugin Helpfulness Verification: Employing a Small Language Model (SLM) for relevance checks, achieving 88% to 98.9% accuracy. 4. Generating Accurate Responses: Evaluating 6 LLMs for answer generation accuracy (81.3% to 93.7%) and implementing automatic prompt tuning. 5. LLM Judge for Quality Assessment: Using LLM judges for scalable response evaluation, achieving 79% to 95% agreement with human judgments.