Chatting Your Way Through 4500 NeurIPS papers with Cerebras - Cerebras

The post details the development of a web application for NeurIPS 2024 papers that uses Cerebras Inference and PostgreSQL's full-text search for RAG. It highlights the challenge of RAG latency bottlenecked by inference speed and describes the process of indexing PDFs into a vector database using LlamaIndex, text extraction, chunking, metadata enrichment, and embedding creation with BAAI's bge-large-en-v1.5 model. The use of Supabase for vector storage and pgvector for similarity search is explained. A key contribution is the reported 17x speed-up in response time by switching from GPU-based providers to Cerebras for LLM inference in the RAG pipeline, reducing average answer time from 8.5 seconds to under 2 seconds.