How We Built a Semantic Highlighting Model for RAG Context Pruning and Token Saving

What this post added

Introduced and open-sourced a Semantic Highlighting model (zilliz/semantic-highlight-bilingual-v1) for RAG context pruning. The model is an encoder-only network based on BGE-M3 Reranker v2, trained to score tokens and aggregate scores to sentence level for relevance filtering. It was trained on over 5 million bilingual samples generated using LLM annotation with reasoning, and evaluated to achieve state-of-the-art performance. The model aims to reduce token usage and improve LLM inference by filtering out irrelevant sentences from retrieved documents.