TechPrufer
Sign inGet started

Blogs›Qdrant›AI Framework Orchestration

AI Framework Orchestration

AI Framework Orchestration

3
posts
2024

Qdrant continues to explore and integrate with AI frameworks to enhance LLM application development. This post discusses the challenges and strategies for using LLMs as judges for evaluating chatbot responses, focusing on cost-effectiveness and real-time assessment. It highlights the use of smaller models for evaluation, the importance of tailored metrics, and techniques for mitigating issues like hallucinations and jailbreaking. The discussion touches upon integrating evaluation into the development process, recommending frameworks like Ragas, Quotient AI, and Arize Phoenix for assessing RAG system performance. It also details common RAG underperformance causes related to data ingestion (chunking strategies), embedding model selection (using MTEB benchmarks and FastEmbed), and retrieval optimization (Precision@k, MRR, NDCG).

2024

Best Practices in RAG Evaluation: A Comprehensive Guide - Qdrant

11/24/2024

This post provides a comprehensive guide to evaluating RAG systems, focusing on accuracy and quality. It details how to test for search precision, recall, contextual relevance, and response accuracy. It introduces and explains the use of Ragas, Quotient AI, and Arize Phoenix frameworks for RAG evaluation. The post also identifies common RAG underperformance issues, including improper data ingestion (emphasizing chunk size, overlap, and strategy), incorrect embedding (recommending MTEB benchmarks and FastEmbed), and unoptimized retrieval procedures (discussing Precision@k, MRR, and NDCG metrics).

The challenges in using LLM-as-a-Judge - Sourabh Agrawal | Vector Space Talks - Qdrant

3/19/2024

This post delves into the practical challenges of using LLMs as judges for evaluating AI chatbot responses. It emphasizes the need for cost-effective evaluation by using smaller, cheaper models instead of expensive ones like GPT-4. The author discusses different scenarios for evaluation: in-the-moment checks for safety (like jailbreaking detection) and post-generation analysis for relevance, hallucinations, and quality. It also highlights the importance of customizing evaluation metrics for specific use cases and the role of experimentation in system improvement. The post introduces UpTrain AI as an LLMOps tool that provides scores and insights for evaluating LLM applications.

DSPy vs LangChain: A Comprehensive Framework Comparison - Qdrant

2/23/2024

This post provides a comprehensive comparison of LangChain and DSPy for building AI applications. It details LangChain's architecture, including its Model I/O, Retrieval, and Composition components, and its use of LCEL. It also introduces DSPy's features like Signatures, Modules, and Optimizers for automated prompt tuning. The post offers guidance on selecting between the two frameworks based on project needs and technical considerations.