TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

AI Data Storage Engine

187 posts · Nov 2019 – Jul 2026

What this trail covers

Milvus 3.0 introduces External Collections, enabling Milvus to build and serve indexes directly on data residing in external data lakes (Parquet, Iceberg, Lance, Vortex) without copying it into Milvus-managed storage. This enables a "Vector Lakebase" architecture, unifying AI data serving with open lake storage. The feature maps external fields to Milvus schemas, defines indexes, and uses a "Refresh" operation to prepare the data for Milvus search and query APIs. Loon (Storage v3) is introduced. This post demonstrates integrating DeepSeek v3-0324 with Milvus for RAG, showcasing the model's advanced reasoning and coding capabilities and providing a practical example of building a RAG pipeline with Milvus Lite and an OpenAI-compatible API.

All Milvus trails
  1. Milvus Snapshots: Point-in-Time Collection Views Without Copying DataIntroduced Milvus Snapshots, a feature that creates point-in-time, read-only views of collections by referencing existing data files instead of copying them. This enables lightweight snapshot creation, fast restores by reusing index files, cross-cluster data movement through metadata URIs, and stable views for external consumers and batch jobs. The implementation leverages the write-once nature of Milvus storage files and records file references and metadata to represent a collection's state at a specific moment. This significantly reduces the cost and time for tasks like rollback, testing, and batch processing.
  2. Announcing Milvus 3.0: Lake-Native Vector Search and a More Powerful Retrieval EngineMilvus 3.0 introduces a lake-native path for indexing and serving vector data directly from object storage using open table formats (Parquet, Lance, Iceberg, Vortex) via External Collections. It enhances the retrieval engine with server-side sorting, aggregation, faceted search, StructArray, and an improved sparse index. Loon (Storage v3) is a new manifest-based columnar storage engine for S3-compatible object storage designed to reduce read amplification for point reads. Snapshots provide point-in-time, read-only views for offline jobs. A Spark DataSource V2 connector is added for batch workflow integration. Online schema evolution (adding/dropping columns) is now supported without downtime. New data model features include nullable vectors, TEXT LOB, TTL, MinHash, and Woodpecker.
  3. How to Recover Milvus Data After etcd Corruption Without a BackupThis post describes a specific, non-standard data recovery procedure for Milvus instances suffering from etcd corruption without a backup. It details the steps involved: starting a clean Milvus instance, recreating the collection schema, copying old MinIO binlogs to a recovery prefix, using the Java SDK's bulkInsert API in backup mode to import candidate segments, and validating each segment before inclusion in the final recovered collection. It also explains why MinIO alone cannot rebuild a collection and the importance of validating segments to avoid duplicates.
  4. Open Tag: An Open-Source Claude Tag for Claude Code and CodexIntroduces Open Tag, an open-source implementation of the Claude Tag pattern, leveraging the MFS (Multi-source File-like Search) system. Details the architecture of Open Tag, highlighting MFS as the core component for agent memory and tools. Explains MFS's design principles: a unified URI interface across heterogeneous sources and a retrieval loop combining search and browse. Demonstrates Open Tag's functionality with examples of local installation, agent-guided configuration, and a real-world PR review. Discusses the engineering problem of enabling agents to search and retrieve context from multiple isolated systems efficiently.
  5. How Force Merge Compaction Nearly Doubled Milvus Search QPSIntroduces and details the 'Force Merge Compaction' feature for Milvus. Explains its mechanism for consolidating small segments into larger ones, contrasting it with standard compaction. Provides experimental results demonstrating significant search QPS improvements (76%-87%) and latency reduction on a 1 million vector dataset. Outlines the conditions under which Force Merge is beneficial and when to avoid it, emphasizing its role as a maintenance action for static, read-heavy collections.
  7. When AI Agents Do the Work, What Do We Lose?This post extends the discussion on AI agents by introducing a four-level autonomy ladder (L1-L4) to frame their development and impact. It specifically elaborates on L2 agents' memory capabilities and the role of systems like Milvus in enabling this by providing a searchable retrieval layer for agent memory. The post also emphasizes the critical need for L4 agents that focus on improving human judgment, arguing that without this, the standards agents operate under will stagnate. It frames the evolution of AI agents not just as task execution but as a loop where human judgment defines standards, agents execute, and feedback improves human learning, which in turn refines the standards.
  8. https://zilliz.com/blog/why-we-built-loon-a-storage-engine-for-ai-data-that-never-stops-changingIntroduces Loon, a novel storage engine built to handle the dynamic nature of AI data. It details the architectural decisions and technical considerations behind creating a storage system that can efficiently manage and adapt to continuously evolving datasets, a common challenge in AI development.
  9. How to Run 25 Million Image Vectors on Under 1GB of Memory in MilvusThis post details how to run 25 million 1280-dimensional vectors on under 1GB of memory in Milvus by leveraging FP16 vector storage, mmap for raw vector data, and aggressive scalar filtering before using the FLAT index. It explains why more advanced ANN indexes like AISAQ and IVF_FLAT were not suitable for this specific memory-constrained workload. The post also provides guidance on interpreting Milvus Sizing Tool estimates and outlines a recipe for applying this pattern to other selective query workloads.
  10. 7 Best Open-Source Tools for Claude Code Context ManagementThis post extends the AI Data Storage Engine capability by detailing how Milvus can be integrated with LLM context management tools to combat 'context defocus'. It explains how vector databases and similarity search, exemplified by Milvus, are crucial for semantic code retrieval and cross-session memory in AI agent workflows. The post maps LLM context management challenges (noisy terminal output, tool outputs, codebase navigation, file reads, model verbosity, memory gaps) to potential solutions and highlights the role of vector databases in providing efficient retrieval of relevant information, thereby improving the quality and performance of LLM interactions.
  11. Claude Context: Reduce Claude Code Token Usage with Milvus-Powered Code RetrievalIntroduces Claude Context, an open-source MCP server for code retrieval that uses Milvus. Details its architecture, including AST-based chunking, Merkle tree for incremental updates, and hybrid retrieval. Highlights performance improvements in token usage and tool calls for AI coding agents.
  12. How RoboBrain Builds Long-Term Robot Memory with MilvusIntroduces RoboBrain, an embodied-intelligence operating system for robot task execution, and details its use of Milvus as the long-term semantic memory tier. Explains how Milvus powers the retrieval of past experiences, exception cases, and operational knowledge through vector similarity search and metadata filtering to improve robot decision-making in dynamic physical environments. Highlights the integration of Milvus with other storage systems for different data types and query patterns.
  13. DeepSeek V4 vs GPT-5.5 vs Qwen3.6: Which Model Should You Use?This post evaluates and compares three large language models (DeepSeek V4, GPT-5.5, and Qwen3.6-35B-A3B) based on practical tests relevant to AI application development, including live information retrieval, concurrency bug debugging, and long-context analysis. It demonstrates how to connect DeepSeek V4 to Milvus for retrieval-augmented generation (RAG) pipelines, providing a concrete example of integrating external knowledge bases with LLMs. The analysis covers model specifications, performance in specific tasks, and considerations for deployment and cost.
  14. How to Fix Hermes Agent's Learning Loop with Milvus 2.6 Hybrid SearchThis post details the integration of Milvus 2.6's hybrid search and tiered storage features with the Hermes AI agent. It explains how hybrid search (combining vector and BM25 full-text search with RRF) addresses the limitations of keyword-only retrieval in Hermes's learning loop, enabling it to connect semantically similar information across sessions. It also highlights how Milvus 2.6's tiered storage allows the retrieval backend to fit on a low-cost VPS, making it practical for personal AI agents. The post provides installation instructions and code examples for setting up Milvus 2.6 standalone and configuring a collection for hybrid search, along with a Python script for performing hybrid searches.
  15. How to Add Long-Term Memory to Anthropic's Managed Agents with MilvusThis post introduces two novel patterns for integrating Milvus with Anthropic's Managed Agents to provide long-term memory. The first pattern enables semantic recall within a single agent session by dual-writing indexed events to a Milvus collection, allowing agents to query their own past decisions and actions using vector search. The second pattern establishes a shared agent memory pool across sessions by partitioning a Milvus collection by tenant, enabling parallel agents to learn from collective experience. The post provides code examples for embedding events, performing session-scoped searches, and utilizing Milvus partition keys for tenant isolation. It also discusses production considerations such as indexing strategy, consistency boundaries, and embedding cost control.
  16. Interview with RaBitQ Authors: The TurboQuant Dispute and Why the Storage Selloff Was a False AlarmThis post details the technical debate surrounding vector quantization, specifically comparing RaBitQ and TurboQuant. It highlights RaBitQ's theoretical optimality and its adoption in Milvus (as IVF_RABITQ) and other systems. The post also critiques TurboQuant's benchmark methodology and claims, emphasizing the importance of fair comparisons and the foundational role of random rotation in RaBitQ. It discusses how vector quantization impacts AI infrastructure costs by enabling efficient compression for vector search and KV cache compression.
  17. We Built Graph RAG Without the Graph DatabaseIntroduces 'Vector Graph RAG', a Python library that enables multi-hop reasoning for RAG systems using only Milvus. It details a novel approach to represent graph structures (entities, relations, passages) within Milvus collections using ID references. The post outlines the four-step retrieval flow: seed retrieval, subgraph expansion (one hop by default), LLM reranking of expanded relations, and answer generation. It highlights the performance benefits of avoiding a separate graph database and iterative LLM calls, achieving competitive recall on multi-hop QA benchmarks.
  18. Harness Engineering: The Execution Layer AI Agents Actually NeedThis post introduces the concept of 'Harness Engineering' as a critical discipline for building reliable autonomous AI agents. It details how Harness Engineering relies on hybrid search (semantic and keyword) for agents to access information from live, evolving knowledge bases. The post outlines specific challenges within Harness Engineering, such as context anxiety and self-evaluation bias, and proposes solutions involving structured documentation, layered architectures, and adversarial agent designs (e.g., Planner, Generator, Evaluator). It highlights the need for retrieval systems that can perform both semantic and exact-match queries concurrently to support agent decision-making.
  19. Beyond the TurboQuant-RaBitQ Debate: Why Vector Quantization Matters for AI Infrastructure CostsThis post details the integration of the RaBitQ vector quantization algorithm into Milvus 2.6. It explains the theoretical underpinnings of RaBitQ, its comparison to TurboQuant, and the engineering challenges overcome to make it production-ready in a distributed system. Key contributions include native integration into Milvus's search engine (Knowhere), dedicated SIMD code paths for x86 and ARM architectures to optimize popcount operations, and pre-computation of auxiliary floating-point parameters to eliminate runtime overhead.
  20. How to Choose the Best Embedding Model for RAG in 2026: 10 Models BenchmarkedThis post benchmarks 10 embedding models across cross-modal retrieval, cross-lingual retrieval, key information retrieval, and dimension compression scenarios. It highlights Gemini Embedding 2 as an all-rounder, Qwen3-VL-2B for cross-modal tasks, and Voyage Multimodal 3.5 or Jina Embeddings v4 for dimension compression. The post details the CCKM benchmark methodology, including hard negative generation for cross-modal and cross-lingual tasks, and the impact of modality gap and semantic understanding on retrieval accuracy. It also discusses the trade-offs of dimension compression for storage costs in vector databases like Milvus.
  21. Vector Database High Availability: How to Build a Milvus Standby Cluster with CDCIntroduces and details Milvus's Change Data Capture (CDC) feature for building a standby cluster, enabling cluster-level high availability. Explains the layered HA strategy (node-level, cluster-level CDC, backup) and provides a hands-on tutorial for setting up primary-standby replication using Milvus CDC and Woodpecker WAL, addressing the critical need for disaster recovery in vector databases.
  22. A Deep Dive into Data Addressing in Storage Systems: From HashMap to HDFS, Kafka, Milvus, and IcebergThis post expands on the AI Data Storage Engine by providing a deep dive into the fundamental principles of data addressing in storage systems. It explains how Milvus, like other systems such as HashMap, HDFS, and Kafka, optimizes data location through computation, caching, and pruning to reduce latency. This provides a foundational understanding of the underlying mechanisms that enable efficient data retrieval in vector databases.
  23. Beyond Naive RAG: Build Smarter Systems with Query Routing and Hybrid RetrievalThis post introduces a four-node architecture for building smarter RAG systems, focusing on query routing, query rewriting, retrieval strategy selection, and minimal-context generation. It advocates for unified hybrid retrieval, combining dense and sparse vectors within a single Milvus collection to simplify operations and improve performance. The post also details stage-by-stage evaluation of RAG pipelines using metrics like F1 score for routing and Recall@K/NDCG for retrieval, enabling faster debugging.
  24. GEO Content at Scale: How to Rank in AI Search Without Poisoning Your BrandThis post details the integration of Milvus with OpenClaw and LLMs to build a production pipeline for Generative Engine Optimization (GEO). It showcases how Milvus is used for knowledge storage, semantic deduplication, and RAG retrieval across two collections (knowledge base and article archive). The pipeline expands seed topics into long-tail queries, semantically deduplicates them using Milvus, and then generates articles using LLMs, grounding the output in verified source documents. The post also highlights the use of Milvus Lite for local prototyping and discusses the feedback loop mechanism where generated articles are written back to Milvus to enrich the deduplication and RAG pools.
  25. Fix RAG Retrieval Errors with CRAG, LangGraph, and MilvusThis post details the implementation of a Corrective Retrieval-Augmented Generation (CRAG) system using LangGraph's middleware pattern and Milvus as the vector database. It addresses common RAG failures such as retrieval mismatch, stale content, and memory contamination by introducing an evaluation and correction step between retrieval and generation. The post highlights Milvus's suitability for CRAG due to its multi-tenant isolation via Partition Key, native hybrid retrieval capabilities (dense, sparse, scalar), and flexible schema support with dynamic JSON fields. It provides a practical example of building the CRAG system, including environment setup, data preparation with a specific Milvus schema, and the core workflow involving retrieval, evaluation, correction, and injection of finalized context into the LLM via a dynamic system prompt. The middleware approach in LangGraph is presented as a more maintainable and debuggable alternative to state graphs for complex CRAG pipelines.
  26. How to Cut Vector Database Costs by Up to 80%: A Practical Milvus Optimization GuideThis post provides a practical guide to reducing vector database costs, focusing on Milvus. It details how to select cost-effective indexes (IVF_SQ8, IVF_PQ, DiskANN) over more expensive ones like HNSW, and explains how to leverage MMap (Milvus 2.3+) and tiered storage (Milvus 2.6+) to drastically reduce memory and local disk usage. The guide also emphasizes the cost implications of embedding dimensionality and provides configuration examples for MMap and tiered storage.
  27. Productionizing Semantic Search: How We Built and Scaled Vector Infrastructure at AirtableAirtable details their experience building and scaling a vector search platform using Milvus for production semantic search. Key contributions include: evaluating and selecting Milvus for its self-hosting, scalability, and multi-tenancy features; designing a partitioning strategy with "one base per partition" and mitigating performance degradation by capping partitions per collection and using multiple collections per cluster; selecting HNSW index for its recall and performance; detailing ingestion and query flows; and addressing operational challenges like deployment via Kubernetes CRD and Milvus operator, observability with infrastructure and service-level metrics, node rotation strategies, cold partition offloading for cost efficiency, and a data recovery plan involving re-embedding commonly used bases.
  28. From PDFs to Answers: Building a RAG Knowledge Base with PaddleOCR, Milvus, and ERNIEThis post details the integration of PaddleOCR for document parsing and ERNIE for answer generation, alongside Milvus for vector storage and hybrid search, to build a RAG-based knowledge base. It elaborates on the data extraction layer using PP-StructureV3, text chunking with sliding window overlap, storing vectors and metadata in Milvus with schema considerations and batch insertion, and implementing hybrid search with RRF fusion for improved retrieval accuracy. Specific code snippets for document parsing, text chunking, and Milvus insertion are provided.
  29. I Built a Stock Monitoring Agent with OpenClaw, Exa, and Milvus for $20/MonthThis post details the integration of Milvus Lite for storing personal trading history and preferences to build a stock monitoring agent. It describes creating three distinct collections ('decisions', 'preferences', 'patterns') with a 1536-dimensional embedding space to store different types of personal data. The post also outlines a memory extraction mechanism that uses an LLM to identify and deduplicate insights from conversations before storing them in Milvus, preventing redundant entries by checking for >92% similarity.
  30. Will Gemini Embedding 2 Kill Multi-Vector Search in Vector Databases?This post analyzes the impact of Google's Gemini Embedding 2 on the necessity and application of multi-vector search in vector databases like Milvus. It differentiates between unified multimodal embeddings and multi-vector retrieval, explaining that Gemini simplifies pipelines for data where all modalities describe the same content. However, it argues that Milvus's multi-vector search remains essential for scenarios involving fundamentally different semantic spaces (biometrics, structured attributes) or combining diverse retrieval goals (keyword precision with semantic recall). The post also suggests a hybrid approach where Milvus collections can house both unified Gemini embeddings and additional specialized vector columns.
  31. How to Build a RAG Pipeline with UltraRAG v2 and MilvusThis post details the integration of Milvus with the UltraRAG v2 framework. It demonstrates how to set up a Milvus standalone instance and integrate it into the UltraRAG pipeline as a retrieval module. The post provides code snippets for modifying the retriever.py file to include Milvus support, specifically the `retriever_init_Milvus` and `retriever_search_Milvus` tools. It also outlines the steps for dataset setup and running the complete RAG pipeline using Milvus for retrieval.
  32. Milvus + RustFS+ Vibe Coding: Build a Lightweight RAG Chatbot from ScratchThis post details the construction of a lightweight RAG chatbot using Milvus and RustFS. It covers the installation of Milvus with an S3-compatible object storage backend (RustFS) via Docker Compose, including configuration adjustments for Milvus. The process of reading, chunking, and embedding Markdown documentation using OpenAI's text-embedding-3-large model is described. Embeddings are then stored in a Milvus collection. Finally, a RAG pipeline is implemented using Milvus for retrieval and OpenAI's GPT-5 for response generation, with a code example for the retrieval and LLM call.
  33. How to Build Multimodal RAG with ColQwen2, Milvus, and Qwen3.5This post details the implementation of a multimodal RAG pipeline using Milvus Lite, ColQwen2 for image-based vector encoding, and Qwen3.5 for generation. It demonstrates how to process PDF pages as images, encode them into multi-vector embeddings using ColQwen2, store these embeddings in Milvus Lite, and retrieve relevant page images based on a query. The retrieval logic uses a MaxSim aggregation strategy to score pages based on the highest-scoring patches for each query token. The retrieved page images are then sent to a multimodal LLM (Qwen3.5) for answer generation, showcasing a practical application for handling documents with tables, charts, and scanned content.
  34. Build a Bestseller-to-Image Pipeline for E-Commerce with Nano Banana 2 + Milvus + Qwen 3.5This post details the integration of Milvus into an e-commerce image generation pipeline. It describes using Milvus for hybrid search to find visually similar bestsellers based on a new product's flat-lay image. The retrieved bestseller images are then used as references for image generation models (Qwen 3.5 for style analysis, Nano Banana 2 for generation). The post highlights how Milvus's retrieval capabilities reduce rework and costs in image generation by leveraging existing high-converting bestseller visuals.
  35. Why AI Agents like OpenClaw Burn Through Tokens and How to Cut CostsThis post details how AI agents, specifically OpenClaw, can burn through tokens due to inefficient search and memory management. It proposes solutions by introducing hybrid search (combining BM25 and vector search) and persistent, editable memory. For hybrid search, it highlights index1 (single-machine, SQLite-based), QMD (single-machine, LLM re-ranking), and Milvus (distributed, team/enterprise scale) as options. For memory, it discusses the need for inspectable and controllable storage beyond framework-dependent solutions.
  36. How to Build Production-Ready AI Agents with Long-Term Memory Using Google ADK and MilvusThis post introduces the integration of Milvus with Google's Agent Development Kit (ADK) to build AI agents with long-term memory. It details how Milvus functions as the backend for ADK's MemoryService, enabling semantic retrieval and hybrid queries for persistent memory. The post provides a practical example of building a technical support agent using Milvus Lite/Standalone and ADK, demonstrating memory storage and retrieval functions, environment setup, and Milvus database initialization with a focus on vector indexing and schema design for AI agent memory.
  37. What If You Could See Why RAG Fails? Debugging RAG in 3D with Project_Golem and MilvusThis post details the integration of Milvus 2.6.8 with Project_Golem to provide a production-ready, scalable solution for visualizing RAG retrieval behavior in 3D. It addresses the limitations of the original Project_Golem (full rebuilds, brute-force search, lack of metadata filtering) by leveraging Milvus for vector storage, indexing, and millisecond-level retrieval. Key contributions include: enabling real-time ingestion via Milvus Streaming Nodes, separating retrieval from visualization for independent scaling, and outlining a future path for incremental visualization updates using UMAP's transform() method and WebSockets. The integration allows for immediate search of newly added documents and sets the stage for hybrid search capabilities within the visualization tool.
  38. GLM-5 vs. MiniMax M2.5 vs. Gemini 3 Deep Think: Which Model Fits Your AI Agent Stack?This post provides a comparative analysis of three new large language models (GLM-5, MiniMax M2.5, and Gemini 3 Deep Think) focusing on their capabilities in coding, deep reasoning, and agentic workflows. It details their performance on specific benchmarks and hands-on tests (3D scene rendering, SVG generation), offering guidance on choosing the right model for different use cases. Crucially, it includes a practical tutorial demonstrating how to integrate GLM-5 (or MiniMax M2.5) with Milvus for building a Retrieval-Augmented Generation (RAG) knowledge base, covering dependency setup, data preparation, and LLM/embedding model configuration.
  39. How to Build Production-Ready Multi-Agent Systems with Agno and MilvusThis post details the integration of Milvus with the Agno framework for building production-ready multi-agent systems. It showcases how Milvus serves as a scalable memory layer for AI agents, providing native support within Agno for connection management, retries, batch writes, and embedding generation. The post demonstrates deploying Milvus standalone with Docker and implementing a knowledge assistant agent using Agno and Milvus, highlighting the step-by-step process and testing the end-to-end flow. It also discusses how retrieval results are shared across agents in Agno's Team mode and the operational benefits of decoupling Agno (conversation/orchestration) from Milvus (knowledge storage).
  40. Why Clawdbot Went Viral — And How to Build Production-Ready Long-Running Agents with LangGraph and MilvusThis post demonstrates the practical application of Milvus as a semantic memory store for long-running AI agents. It details how Milvus, specifically Milvus Lite for local development and distributed Milvus for production, can be used to embed and retrieve progress records and git commits, providing agents with persistent, cross-session knowledge. The post highlights Milvus's capabilities in handling billions of vectors, supporting hybrid search for combining vector similarity with traditional filtering, and its role in enabling agents to recall relevant past information for complex tasks.
  41. Why Claude Code Feels So Stable: A Developer’s Deep Dive into Its Local Storage DesignThis post details the local storage design of Claude Code, focusing on its stability and reliability for AI coding assistance. It emphasizes project isolation, immediate data persistence to disk, ordered action history, and easy rollback of code edits. The storage architecture includes a global configuration file (~/.claude.json) and a main data directory (~/.claude/) containing project-specific session data, history, skills, plugins, and file edit history. Configuration is managed through a three-layer model: global, local, and project-level settings, with project-level settings taking the highest priority. This design ensures that Claude Code behaves like a robust local development tool by treating coding sessions as persistent, version-controlled projects rather than ephemeral chat interactions.
  42. Is RAG Becoming Outdated Now That Long-Running Agents Like Claude Cowork Are Emerging?This post analyzes the evolving landscape of AI agent memory, contrasting traditional RAG with emerging long-running agent capabilities like Claude Cowork. It details how Claude Cowork uses a hybrid approach of persistent user memory and on-demand conversational retrieval (semantic and keyword search) to maintain state across long sessions, differentiating it from RAG's knowledge-driven retrieval. The post also contrasts this with ChatGPT's more structured memory system (User Memory, Session Metadata, cross-session summaries). It highlights the challenges of implementing writable agent memory, including deciding what to remember, how to store and tier memory, and how to write fast without breaking retrieval. The core technical contribution is the analysis of these memory management strategies and their implications for vector databases in AI agent development.
  43. Building AI Agents in 10 Minutes Using Natural Language with LangSmith Agent Builder + MilvusThis post demonstrates how to build an AI agent using LangSmith Agent Builder and Milvus. It highlights the use of Milvus as a vector database for providing memory and context to AI agents, enabling them to answer technical questions and remember user preferences. The post details the steps involved in creating an agent, configuring tools (including Milvus's MCP server for vector search), and testing its functionality. It also contrasts Agent Builder with DeepAgents, emphasizing Agent Builder's focus on simplicity and rapid prototyping for AI agent development with integrated memory capabilities.
  44. A Brief Introduction to the ScaNN IndexIntroduces the ScaNN index, detailing its two key innovations: a score-aware quantization objective that preserves the relative ordering of true neighbors and FastScan, a SIMD-optimized 4-bit PQ lookup path for reduced memory bottlenecks. Provides experimental results demonstrating ScaNN's performance advantages over IVFFLAT and IVF_PQ in terms of QPS and memory footprint, particularly when raw data is not loaded.
  45. How We Built a Semantic Highlighting Model for RAG Context Pruning and Token SavingIntroduced a new open-source Semantic Highlighting model (zilliz/semantic-highlight-bilingual-v1) based on BGE-M3 Reranker v2. This model performs token-level relevance scoring to identify and highlight semantically relevant sentences within retrieved documents for RAG context pruning. It was trained on over 5 million bilingual samples using a reasoning-based LLM annotation pipeline with Qwen3 8B and vLLM for efficient data generation. The model aims to reduce token costs, improve answer quality, and enhance interpretability in RAG systems.
  46. MinIO Stops Accepting Community Changes: Evaluating RustFS as a Viable S3-Compatible Object Storage Backend for MilvusThis post evaluates RustFS as an S3-compatible object storage backend for Milvus, exploring its architecture and compatibility. It provides a hands-on guide to replacing MinIO with RustFS in a Milvus deployment and discusses experimental storage performance testing.
  47. Reflections on ChatGPT and Claude’s Memory Systems: What It Takes to Enable On-Demand Conversational RetrievalThis post details how Milvus 2.6's hybrid dense-sparse retrieval, efficient scalar filtering, and tiered storage enable on-demand conversational retrieval for AI agents. It contrasts ChatGPT's precomputed context injection with Claude's RAG-style, on-demand retrieval, highlighting the architectural constraints (low latency tolerance, hybrid search, storage-compute separation) that make the latter practical with systems like Milvus.
  48. We Trained and Open-Sourced a Bilingual Semantic Highlighting Model for Production RAG and AI SearchThis post introduces and open-sources a new bilingual semantic highlighting model designed for production RAG and AI search. The model addresses limitations of existing solutions like OpenSearch's semantic highlighter (small context window, poor out-of-domain generalization) and Provence/X Provence (English-only or limited multilingual performance, restrictive licensing). The new model aims for strong multilingual performance, a large context window, robust out-of-domain generalization, high precision, and a permissive license (MIT or Apache 2.0). It is trained to identify semantically aligned text spans, improving the explainability of search results in RAG pipelines and AI agents.
  49. Phrase Match with Slop in Milvus 2.6: How to Improve Phrase-Level Full-Text Search AccuracyIntroduces and explains the new Phrase Match feature in Milvus 2.6, which enables full-text search with positional indexing and a 'slop' parameter for flexible phrase matching. Details the technical implementation using Tantivy, the schema configuration required for VARCHAR fields (`enable_analyzer`, `enable_match`), and provides examples of how to use `PHRASE_MATCH` with varying `slop` values to filter search results before vector similarity ranking.
  50. Embedding First, Then Chunking: Smarter RAG Retrieval with Max–Min Semantic ChunkingIntroduces Max–Min Semantic Chunking, a new strategy for text chunking in RAG pipelines. This method embeds all sentences first and then uses semantic similarity (cosine similarity) to determine chunk boundaries, aiming for more meaningful and coherent chunks than traditional fixed-length or recursive methods. It details the algorithm's steps, including measuring chunk consistency and comparing new sentences to existing chunks, and discusses its strengths (dynamic boundaries, practical tuning, low overhead) and limitations (potential for context fragmentation in long documents).
  51. Keeping AI Agents Grounded: Context Engineering Strategies that Prevent Context Rot Using MilvusThis post details strategies for preventing context rot in long-running AI agent conversations by engineering the context provided to LLMs. It explains the causes of context rot (Transformer O(n^2) complexity, out-of-distribution training data) and presents three main approaches: Just-in-Time (JIT) retrieval, pre-retrieval using vector search (specifically Milvus), and a hybrid approach combining both. The post highlights Milvus's capabilities in supporting hybrid retrieval with vector search and scalar filtering, and its support for incremental updates. It provides guidance on choosing the right approach based on knowledge stability and task predictability.
  52. LangChain 1.0 and Milvus: How to Build Production-Ready Agents with Real Long-Term MemoryThis post details how Milvus can be integrated with LangChain 1.0 to provide persistent, intelligent memory for agents. It explains how Milvus's vector database capabilities enable efficient storage, retrieval, and semantic similarity search of information, going beyond LangChain's core reasoning and tool-calling functionalities. The post highlights Milvus as a scalable and high-performance memory layer for production-ready agents.
  53. Introducing the Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent WorkloadsIntroduces the Ngram Index for Milvus, enabling faster LIKE queries and keyword matching by indexing n-grams of text. Details the index construction (min_gram, max_gram) and query acceleration phases, including handling of infix and wildcard matches. Discusses limitations like increased index size and evaluates performance benchmarks.
  54. Stop Paying for Cold Data: 80% Cost Reduction with On-Demand Hot–Cold Data Loading in Milvus Tiered StorageIntroduces and details Milvus 2.6's Tiered Storage architecture, which enables on-demand loading of hot and cold data. Explains the inefficiencies of the previous full-load model and how Tiered Storage addresses them through lazy loading, partial loading (tenant-aware vector index loading, column-level scalar field loading), and LRU cache eviction. Provides performance and cost-saving evaluations.
  55. Introducing AISAQ in Milvus: Billion-Scale Vector Search Just Got 3,200× Cheaper on MemoryIntroduces AISAQ, a new disk-based vector index for Milvus that achieves a 'Zero-DRAM-Footprint Architecture' by storing PQ codes and other search-critical data on disk. This significantly reduces memory usage for billion-scale vector search workloads, offering a 3,200x reduction in memory footprint compared to previous approaches while maintaining performance. The post explains the underlying principles of graph-based vector search, the trade-offs involved, and how AISAQ builds upon DISKANN by optimizing disk data layout for efficient I/O.
  56. Optimizing NVIDIA CAGRA in Milvus: A Hybrid GPU–CPU Approach to Faster Indexing and Cheaper QueriesIntroduces and details the hybrid GPU-CPU approach for CAGRA indexing in Milvus 2.6.1. Explains the CAGRA algorithm (NN-Descent and 2-hop pruning) and its GPU acceleration. Details the `adapt_for_cpu` parameter and its four execution modes for balancing GPU construction with CPU querying. Presents experimental results demonstrating significant index build time improvements (12-15x) and discusses query performance and recall accuracy.
  57. Bringing Geospatial Filtering and Vector Search Together with Geometry Fields and RTREE in Milvus 2.6Introduces the Geometry Field (DataType.GEOMETRY) for storing spatial data (Point, LineString, Polygon) using WKT format. Implements an RTREE index for efficient spatial filtering through coarse and fine filtering stages, supporting operators like st_contains, st_within, and st_dwithin. Enables joint retrieval by combining RTREE-based spatial filtering with vector similarity search, allowing for spatially aware intelligent retrieval.
  58. Unlocking True Entity-Level Retrieval: New Array-of-Structs and MAX_SIM Capabilities in MilvusIntroduces and explains Milvus's new Array of Structs field type, which allows a single record to store an ordered list of Struct elements, each capable of holding multiple vectors and scalar fields. Details the new MAX_SIM scoring metric, which computes entity-level similarity by taking the maximum similarity score between query vectors and all vectors within an Array of Structs field for a given entity. Demonstrates how these features together enable true entity-level retrieval, solving issues like fragmented search results and duplicate entities in RAG, e-commerce, and other multi-vector applications.
  59. JSON Shredding in Milvus: 88.9x Faster JSON Filtering with FlexibilityIntroduces JSON Shredding, a new feature that optimizes the storage and querying of semi-structured JSON data in Milvus. It categorizes JSON keys into typed, dynamic, and shared, storing them in optimized columnar formats (Arrow/Parquet) or a compact binary-JSON column with an inverted index for shared keys. This enables schema-free agility with high query performance, demonstrated by benchmarks showing up to 88.9x speedup for shared key queries and 15-30x for typed key queries.
  60. Introducing the Embedding Function: How Milvus 2.6 Streamlines Vectorization and Semantic SearchThis post details the introduction of the Data-in, Data-out feature (Embedding Function) in Milvus 2.6. It describes the underlying 'Function' module that enables Milvus to invoke external embedding providers (OpenAI, AWS Bedrock, Google Vertex AI, Cohere, Hugging Face) directly during data ingestion and query time. The post outlines the six-step workflow: input raw data, generate embeddings via the configured provider, store embeddings, submit raw queries, perform semantic search using the same model, and return results. It also provides configuration details for `milvus.yaml`, schema definition with `TEXTEMBEDDING` function type, and usage examples for inserting raw data and performing text-based semantic searches.
  61. 7 Years, 2 Major Rebuilds, 40K+ GitHub Stars: The Rise of Milvus as the Leading Open-Source Vector DatabaseThis post chronicles the 7-year journey of Milvus, highlighting two major rebuilds and its growth to over 40,000 GitHub stars. It details key milestones including the open-sourcing of Milvus 0.10, joining the LF AI & Data Foundation, winning the BigANN challenge, and the ground-up reinvention for Milvus 2.0. It then focuses on the significant advancements in 2025 with Milvus 2.5 introducing native hybrid search and Milvus 2.6 delivering substantial performance and cost improvements through features like tiered storage, 1-bit quantization (RaBitQ), an improved full-text engine, Path Index for JSON, AiSAQ, Geospatial Search with R-Tree, CAGRA+ Vamana, a streamlined data workflow, and increased collection support. The post also introduces two new open-source tools: DeepSearcher for local deep research and Claude Context for enhanced AI coding assistants. Finally, it emphasizes Milvus's adoption by over 10,000 teams in production for various AI workloads, citing performance metrics like sub-50ms retrieval and multi-tenant capabilities.
  62. Choosing a vector database for ANN search at RedditThis post details Reddit's process for selecting a vector database for ANN search, evaluating Milvus 2.4 alongside other solutions. It highlights Milvus's strengths in hybrid search, metadata filtering, support for multiple indexing methods, and its open-source nature as key factors in their decision. The evaluation process involved qualitative and quantitative assessments against functional and non-functional requirements, emphasizing scale, reliability, community, and expressive metadata types.
  63. Multimodal RAG Made Simple: RAG-Anything + Milvus Instead of 20 Separate ToolsThis post details the integration of RAG-Anything with Milvus to create a unified multimodal RAG system. It explains how RAG-Anything's "1 + 3 + N" architecture (core engine, 3 modal processors, N parsers) handles diverse content types in parallel. Milvus is used as the vector database to store and search across various embedding types generated from text, images, tables, and equations, simplifying the infrastructure by replacing multiple specialized tools with a single, scalable vector search solution. A hands-on demo with Milvus Lite, Alibaba Cloud API, and specific models (qwen-vl-max, tongyi-embedding-vision-plus) is provided, along with project structure, dependencies, and configuration details.
  64. OpenAgents x Milvus: How to Build Smarter Multi-Agent Systems That Share MemoryThis post details the integration of Milvus with the OpenAgents framework to provide a scalable, high-performance long-term memory layer for multi-agent systems. It highlights how Milvus enables semantic search, billion-scale horizontal scalability, and multi-tenant isolation through Partition Key for agent memory. The post also provides a practical example of building a developer-support community chatbot using OpenAgents and Milvus, demonstrating how agent conversations, past solutions, and user queries are stored and retrieved via vector embeddings.
  65. Gemini 3 Pro + Milvus: Building a More Robust RAG With Advanced Reasoning and Multimodal PowerThis post details the integration of Milvus with Google's Gemini 3 Pro for building advanced RAG systems. It highlights Gemini 3 Pro's multimodal understanding, code generation, AI agent capabilities, and generative UI features. The post provides a practical guide on setting up a RAG pipeline using Milvus Lite for local development, including data preparation from Milvus documentation, embedding generation using `models/text-embedding-004`, and inserting data into a Milvus collection. It demonstrates how Milvus acts as the retrieval backbone, grounding Gemini 3 Pro's responses in enterprise data. The tutorial covers prerequisites, dataset preparation, LLM and embedding model setup, and the workflow for retrieving relevant data using similarity search.
  66. Beyond Context Overload: How Parlant × Milvus Brings Control and Clarity to LLM Agent BehaviorThis post introduces Parlant, an agent framework that uses Alignment Modeling with Granular Guidelines. It details how Milvus is integrated to perform semantic retrieval of these guidelines, enabling dynamic context injection for LLM agents. The post provides code examples for defining guidelines and interacting with Milvus for guideline matching and user memory storage, highlighting Milvus's role in achieving low-latency, high-accuracy retrieval for controlled and explainable agent behavior.
  67. Drag, Drop, and Deploy: How to Build RAG Workflows with Langflow and MilvusThis post details the integration of Langflow, a visual RAG workflow builder, with Milvus, a vector database. It provides a step-by-step guide on deploying Milvus, setting up Langflow, and configuring a RAG template to use Milvus as the vector store. The guide covers preparing test data, ingesting it into Milvus via Langflow, and running the full RAG workflow. It highlights the code-light approach to building RAG applications and the benefits of combining Langflow's visual design with Milvus's high-performance search capabilities.
  68. Understanding IVF Vector Index: How It Works and When to Choose It Over HNSWThis post provides a deep dive into the Inverted File (IVF) vector index, a core component for Approximate Nearest Neighbor (ANN) search in vector databases. It explains the IVF index's architecture, including clustering, centroids, and inverted lists. It details the index building process involving K-means clustering, vector assignment, and optional compression techniques like Scalar Quantization (SQ8) and Product Quantization (PQ). The post also outlines the search process, emphasizing the role of nprobe in balancing recall and latency. Practical advice is given on choosing nlist and tuning nprobe for different scales and performance requirements, along with a comparison of IVF variants (IVF_FLAT, IVF_PQ, IVF_SQ8).
  69. vLLM Semantic Router + Milvus: How Semantic Routing and Caching Build Scalable AI Systems the Smart WayThis post introduces the integration of Milvus as a semantic cache layer with the vLLM Semantic Router. It details how Milvus's distributed vector database capabilities enable efficient semantic caching for AI systems by storing and retrieving embeddings, thereby improving scalability, reducing latency, and lowering costs. The post explains the architecture of semantic routing, mixture of models, and the cache layer, emphasizing Milvus's role in handling large-scale semantic search with ANN algorithms. It also provides a practical guide for developers to test this integration locally, demonstrating the caching mechanism for AI applications.
  70. Tokenize Smarter, Retrieve Better: A Deep Dive into Milvus Analyzer for Full-Text SearchIntroduces and details the Milvus Analyzer, a new component for text preprocessing and tokenization that enables efficient full-text search. Explains the Tokenizer and Filter components, various Analyzer types (Standard, English, Chinese, Multi-language, Custom), and provides Python SDK examples for their usage. This enhances Milvus's hybrid search capabilities by improving keyword matching.
  71. Smarter Retrieval for RAG: Late Chunking with Jina Embeddings v2 and MilvusThis post introduces and demonstrates a 'Late Chunking' strategy for RAG. This strategy involves embedding the entire document first using a long-context transformer (Jina Embeddings v2) and then averaging contiguous token embeddings to form chunk vectors. This contrasts with traditional methods that pre-chunk text before embedding. The post provides Python code for implementing this strategy and shows how to store and query the resulting chunk embeddings in Milvus. It includes a comparative analysis of similarity scores between Late Chunking and traditional chunking on a Milvus release note example, showing higher scores for Late Chunking. The post also validates Milvus's search accuracy by comparing its retrieval results to brute-force cosine similarity calculations.
  72. From Word2Vec to LLM2Vec: How to Choose the Right Embedding Model for RAGThis post provides a comprehensive guide on evaluating embedding models for RAG systems, detailing eight key factors: context window, tokenization unit, dimensionality, vocabulary size, training data, cost, MTEB score, and domain specificity. It also discusses additional perspectives like multilingual alignment and adversarial testing. While not directly adding new Milvus features, it significantly enhances the understanding of how to best leverage Milvus within a RAG architecture by guiding users on selecting optimal embedding models, which directly impacts the effectiveness of Milvus's vector search capabilities in RAG applications.
  73. When Context Engineering Is Done Right, Hallucinations Can Be the Spark of AI CreativityThis post introduces the concept of 'Context Engineering' for AI systems, framing hallucinations not just as defects but as potential sparks of creativity. It details three pillars of context engineering: Instructions, Knowledge, and Tools. The post also analyzes challenges associated with long context windows (poisoning, distraction, confusion, clash) and proposes six strategies to manage them: Context Isolation, Pruning, Summarization, Offloading, Strategic RAG, and Optimized Tool Loading. Milvus is specifically mentioned as a component for the 'Knowledge Layer' and for 'Context Offloading' and 'Strategic RAG', highlighting its role in providing accessible, externalized knowledge for AI models.
  74. Nano Banana + Milvus: Turning Hype into Enterprise-Ready Multimodal RAGThis post demonstrates a multimodal RAG pipeline by integrating Milvus with the Nano Banana text-to-image generation model and CLIP for embedding. It provides a practical example of setting up a Milvus client, loading the CLIP model, defining image and text encoding functions, creating a Milvus collection with dynamic fields and cosine similarity, processing and inserting image embeddings, and performing text-to-image similarity searches. The post also outlines the conceptual integration of Nano Banana for generating brand promotional images based on retrieved visual assets, highlighting Milvus's role in enabling enterprise-ready multimodal RAG.
  75. Why I’m Against Claude Code’s Grep-Only Retrieval? It Just Burns Too Many TokensIntroduces Claude Context, an open-source MCP plugin that adds semantic code search capabilities to AI coding assistants. It details the system architecture, including the use of Zilliz Cloud (Milvus) as the vector database and support for various embedding models. Key technical challenges addressed include intelligent code chunking using AST-based parsing and LangChain's text splitter, and efficient handling of code changes through a Merkle Tree-based synchronization mechanism. The post highlights the benefits of this approach over grep-only methods, such as reduced token usage and improved search accuracy.
  76. AI Agents or Workflows? Why You Should Skip Agents for 80% of Automation TasksThis post demonstrates the integration of Milvus with Refly and MCP-Milvus-Server to build AI-powered content creation workflows. It details the setup process for Milvus, Refly, and the MCP server, and provides examples of using natural language to query Milvus via MCP-Milvus-Server. It also illustrates building a content deployment guide using Refly's workflow orchestration, highlighting multi-modal capabilities.
  77. GPT-oss vs o4-mini: Edge-Ready, On-Par Performance — Dependable, Not Mind-BlowingThis post demonstrates building a reasoning-capable RAG system using GPT-oss-20b and Milvus. It provides a hands-on tutorial covering environment setup (pymilvus, openai, requests, tqdm), dataset preparation using Milvus documentation, model setup with OpenAI's GPT-oss models (accessed via OpenRouter or local deployment), and Milvus vector database setup (MilvusClient for local or Zilliz Cloud). It details adding documents to Milvus by creating embeddings and inserting them, and outlines the RAG query pipeline by performing semantic search and retrieving top results. The post also discusses the technical specifications and performance comparisons of GPT-oss models, highlighting their MoE architecture, edge-readiness, and cost-efficiency.
  78. Build a Production-Grade Chatbot with Kimi K2 and MilvusThis post details the implementation of a production-grade chatbot using Kimi K2 and Milvus. It provides a practical example of integrating a large language model (Kimi K2) with a vector database (Milvus) for RAG applications. The core contribution is a Python implementation demonstrating automatic file processing, chunking, semantic search using OpenAI embeddings, and intelligent decision-making orchestrated by the LLM. It includes code for a `VectorDatabase` class handling Milvus connections, collection creation, document insertion, and similarity search, and a `SmartAssistant` class orchestrating tool usage. The post highlights the benefits of Kimi K2's agent capabilities and API compatibility, and the ease of use of Milvus for storing and querying vector embeddings.
  79. Building an Open-Source Alternative to Cursor with Code ContextIntroduces Code Context, an open-source semantic code search engine that leverages Milvus for vector storage. Details its architecture, including AST-based code chunking, live incremental indexing, and integration with AI coding agents via the MCP protocol. Provides examples for integrating with existing tools (Claude Code, VSCode) and building custom AI assistants using Milvus and embedding models. Highlights private deployment capabilities and real-time indexing mechanisms.
  80. Hands-on RAG with Qwen3 Embedding and Reranking Models using MilvusThis post details the integration of external Qwen3 embedding and reranking models with Milvus for a Retrieval-Augmented Generation (RAG) system. It provides a practical guide on setting up the environment, preparing data, initializing embedding and reranking models, and loading data into Milvus. The core contribution is demonstrating a two-stage retrieval pipeline: initial dense retrieval using Qwen3 embeddings (with query-specific prompting) followed by precision refinement using Qwen3 reranking models, and finally generation with GPT-4. It showcases how to leverage Milvus for storing and searching vector embeddings generated by external models, and how to implement a hybrid search strategy combining dense retrieval with reranking for improved RAG performance.
  81. From Docs to Dialogue: Building a Production-Ready AI Assistant with Spring Boot and MilvusThis post details the practical implementation of a RAG-based AI assistant using Spring Boot and Milvus. It covers document ingestion pipeline using TikaDocReader and text splitting, generating embeddings with Ollama's Nomic Embed Text model, and storing them in Milvus. It demonstrates integrating Milvus for semantic search within a Spring AI RAG pipeline, including conversation context management using Redis and streaming responses via SSE. The post also provides examples for securing the API with Spring Security (RBAC) and implementing full-stack observability using OpenTelemetry JavaAgent for tracing and Micrometer for Prometheus metrics, specifically highlighting the need for the JavaAgent due to Milvus's gRPC usage.
  82. Introducing Milvus 2.6: Affordable Vector Search at Billion ScaleMilvus 2.6 introduces RaBitQ 1-bit quantization for significant memory reduction (up to 72%) and performance gains (4x QPS) without compromising recall. It also enhances full-text search with a revamped multi-language analyzer, adds Phrase Match for nuanced word order search, and introduces Time-Aware Decay Functions for relevance ranking based on age. A new Function interface integrates embedding models directly into the data pipeline for a streamlined 'Data-In, Data-Out' experience. The architectural evolution includes replacing Kafka/Pulsar with a new Woodpecker WAL system for improved performance and reduced operational complexity, and introduces tiered storage for cost reduction.
  83. No Python, No Problem: Model Inference with ONNX in Java, or Any Other LanguageThis post details how to export HuggingFace models (specifically text embedding models) into the ONNX format and then perform inference using these ONNX models directly from Java. It covers exporting both the model and its tokenizer using Python libraries like `sentence-transformers` and `onnxruntime-extensions`, and then demonstrates how to load and run these ONNX files within a Java application using the ONNX Runtime Java API. This enables local, language-agnostic model inference for AI applications, particularly relevant for data ingestion into vector databases like Milvus.
  84. Why AI Databases Don't Need SQLThis post argues for the obsolescence of SQL in AI applications and positions vector databases like Milvus as the future. It contrasts SQL's limitations with unstructured data and semantic search against the capabilities of vector databases, highlighting Milvus's native multimodal support, agent-friendly architecture, and semantic intelligence. It also presents benchmark data showing Milvus's superior performance over PostgreSQL with pgvector for vector similarity search.
  85. We Benchmarked 20+ Embedding APIs with Milvus: 7 Insights That Will Surprise YouThis post benchmarks the latency of over 20 embedding APIs across different geographic regions and network conditions, revealing that network latency and provider-specific optimizations are critical performance factors in RAG systems. It highlights that popular embedding APIs are not always the fastest, and that local inference can be a competitive alternative. The post also validates that Milvus's TextEmbedding Function adds negligible overhead to the overall operation.
  86. Understanding Hierarchical Navigable Small Worlds (HNSW) for Vector SearchThis post provides a detailed explanation of the Hierarchical Navigable Small Worlds (HNSW) algorithm, a key indexing strategy for vector databases. It covers the underlying Navigable Small Worlds (NSW) concept, the construction and search processes for both NSW and HNSW, and the hierarchical modifications that improve search efficiency and convergence. The post emphasizes HNSW's role in enabling efficient similarity search for large-scale vector datasets and links to a pedagogical Python implementation.
  87. DiskANN ExplainedIntroduces and explains the DiskANN index type for Milvus. Details its architecture, how it leverages SSDs for large-scale vector similarity search, and its trade-offs compared to in-memory methods. Explains the Vamana algorithm for graph construction and the use of product quantization for efficient search guidance. Provides Python code examples for creating collections with the DiskANN index and mentions tunable parameters.
  88. Milvus 2.6 Preview: 72% Memory Reduction Without Compromising Recall and 4x Faster Than ElasticsearchMilvus 2.6 introduces RaBitQ 1-bit quantization with adjustable Refine optimization (SQ4/SQ6/SQ8) for significant memory reduction and performance improvements. It also enhances full-text search with a revamped multi-language analyzer (Lindera, ICU, enhanced Jieba), adds Phrase Match with a 'slop' parameter for word order control, and introduces Time-Aware Decay Functions (exponential, Gaussian, linear) for relevance ranking based on document age. A new Function interface allows direct integration of embedding models (OpenAI, AWS Bedrock, Google Vertex AI, Hugging Face) for streamlined data ingestion and vectorization. Architecturally, a hot-cold tiered storage system is introduced for cost-effective scaling, and a new Streaming Node enables real-time vector processing with direct integration to streaming platforms.
  89. MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training DataIntroduces and details the implementation of MinHash LSH indexing in Milvus 2.6 for approximate data deduplication. Explains the underlying concepts of MinHash (shingling, hashing) and LSH (banding technique) for efficient near-duplicate detection. Describes the native integration within Milvus, including the MINHASH_LSH index type, JACCARD metric, and distributed architecture support. Provides practical guidance on generating MinHash signatures and using Milvus SDK for schema creation, index building, and data insertion for deduplication tasks.
  90. Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQIntroduces and details the integration of RaBitQ, a novel 1-bit vector compression technique, into Milvus. Explains the engineering challenges and tradeoffs in implementing RaBitQ within Milvus's Knowhere search engine, including pre-computation of auxiliary data and hardware acceleration using CPU instructions (VPOPCNTDQ for AVX512). Describes query optimization techniques like scalar quantization on query vectors and refinement. Introduces the new `IVF_RABITQ` index type in Milvus 2.6, which combines RaBitQ with IVF clustering. Provides usage examples and benchmarking results showing a 3x QPS increase with comparable accuracy.
  91. Vector Search in the Real World: How to Filter Efficiently Without Killing RecallThis post details how Milvus and Zilliz Cloud implement optimizations for efficient metadata filtering during vector searches. It covers graph index optimization using the Alpha strategy and dynamic fallback to brute-force scans for highly selective filters. It also introduces metadata-aware indexing with specialized subgraphs (column graphs) and iterative filtering inspired by VBase, which processes data in batches to reduce filter operation costs. Additionally, it describes external filtering, a client-side SDK solution for hybrid data systems that reverses the traditional workflow by performing vector search first and then applying custom filters to batches of candidates.
  92. Hands-on with Qwen 3 and Milvus: Building RAG with the Latest Hybrid Inference ModelsThis post details building a local, cost-aware Retrieval-Augmented Generation (RAG) system by integrating the Qwen 3 hybrid inference models with Milvus. It provides hands-on code for setting up the environment, preparing data from Milvus documentation, configuring Milvus Lite for local storage, and using the OpenAI-compatible API for Qwen 3 models. The tutorial covers creating a Milvus collection, generating embeddings for text chunks using `pymilvus.model.DefaultEmbeddingFunction`, inserting data, and performing semantic searches to build the RAG query system. It demonstrates how to retrieve relevant documents for a given question and outlines the process of using these retrieved documents to augment the Qwen 3 model's response.
  93. Hands-on Tutorial: Build a RAG-Powered Document Assistant in 10 Minutes with Dify and MilvusThis post demonstrates a practical implementation of a RAG-powered document assistant by integrating Milvus with Dify. It details the steps involved in setting up Dify and Milvus containers, configuring OpenAI API keys, inserting documents into a knowledge base (which involves chunking and embedding), and creating the RAG application. The post highlights how Milvus acts as the vector database for storing document embeddings, enabling efficient similarity search for retrieval-augmented generation.
  94. Full-Text Search in Milvus - What's Under the HoodIntroduces and details the implementation of Sparse-BM25 full-text search in Milvus 2.5. Explains the BM25 algorithm and its transformation into an Inner Product calculation for sparse vectors. Details Milvus's novel approach using dynamic statistics (avgdl, term distribution) and search-time encoding to handle the complexities of BM25 computation internally, removing client-side burden. Provides a PyMilvus example for defining a collection with a text field and a BM25 function, creating an AUTO_INDEX with BM25 metric, inserting raw text, and performing searches.
  95. How to Choose the Right Embedding Model?This post provides a practical framework for selecting embedding models, detailing steps such as defining task and business requirements, evaluating data characteristics (modality, domain specificity, embedding type), researching available models (popularity, licensing), evaluating candidate models (accuracy, benchmark performance, load testing), planning model integration (weights selection, self-hosting vs. third-party inference, API design), and conducting end-to-end testing (performance, robustness). It emphasizes the importance of aligning model choice with specific use cases and operational constraints.
  96. Parsing is Hard: Solving Semantic Understanding with Mistral OCR and MilvusThis post demonstrates the integration of Mistral OCR with Milvus for semantic document understanding and search. It showcases how Mistral OCR can extract text and structure from complex documents (handwritten notes, invoices) and how this extracted information can be inserted into Milvus for vector search. The post highlights Milvus's capabilities in handling large-scale vector data and its support for full-text search, making it suitable for RAG systems.
  97. Generating More Creative and Curated Ghibli-Style Images with GPT-4o and MilvusThis post details the integration of Milvus with GPT-4o for enhanced image generation. It provides a practical example of using Milvus as a vector database to store and search image embeddings extracted using ResNet-50. The workflow involves preparing data, defining a feature extractor, creating a Milvus collection, inserting image embeddings, and performing similarity searches. The results of these searches are then used as context for GPT-4o prompts to generate creative, curated images. The post also offers tips for effective image generation with GPT-4o.
  98. DeepSeek V3-0324: The "Minor Update" That's Crushing Top AI ModelsThis post introduces a practical RAG pipeline using Milvus Lite and the DeepSeek v3-0324 LLM. It details the steps for setting up the environment, preparing data from Milvus documentation, configuring Milvus collections with Milvus Lite, embedding documents using Milvus's DefaultEmbeddingFunction, and performing similarity searches. The core contribution is demonstrating how to combine the advanced reasoning capabilities of DeepSeek v3-0324 with Milvus's vector database for efficient knowledge retrieval and synthesis in a RAG system. It highlights the use of an OpenAI-compatible API for DeepSeek and the integration with Milvus's embedding capabilities.
  99. What Exactly is a Vector Database and How Does It WorkThis post provides a foundational overview of vector databases, explaining their core concepts, mathematical underpinnings (vector spaces, embeddings, distance metrics), and architectural components (storage, index, query, service layers). It details the vector search workflow from data transformation to retrieval and touches upon consistency challenges in distributed systems.
  100. Stop Building Vanilla RAG: Embrace Agentic RAG with DeepSearcherThis post introduces DeepSearcher, an open-source project that implements agentic RAG, inspired by OpenAI's Deep Research. It details the shift from traditional RAG to agentic RAG, highlighting the latter's proactive, multi-step, and iterative approach to problem-solving. The architecture of DeepSearcher is presented, consisting of a Data Ingestion Module (leveraging Milvus for vector storage) and an Online Reasoning and Query Module that implements diverse agent strategies. Use cases like overview-type queries are illustrated with examples of DeepSearcher's iterative query generation and report synthesis. The post emphasizes the transparency provided by logging search and reasoning steps, which is crucial for debugging and optimizing RAG systems.
  101. Why Manual Sharding is a Bad Idea for Vector Database And How to Fix ItThis post elaborates on Milvus's automated sharding and scaling mechanisms, contrasting them with the drawbacks of manual sharding in vector databases. It details Milvus's segment-based architecture, two-layer routing, and query processing flow to explain how it achieves seamless scalability without operational burden. It also mentions the upcoming Milvus 3.0 feature of dynamic shard splitting.
  102. A Day in the Life of a Milvus DatumThis post provides a detailed walkthrough of the internal data flow for insert and search operations in Milvus. It elaborates on the lifecycle of a datum from insertion through proxy nodes and message queues to data nodes, segments, and chunks in persistent storage. It also describes the processes of segment sealing, merging, compaction, and index building by index nodes, and how query nodes retrieve and process data for search. The key takeaway is the separation of concerns across node types and the separation of storage and compute, enabling independent scaling.
  103. AI for Smarter Browsing: Filtering Web Content with Pixtral, Milvus, and Browser UseThis post demonstrates a practical application of Milvus within an AI-powered content filtering system. It showcases the integration of Milvus with Browser-Use for web scraping and Pixtral (a vision-language model) for content analysis and classification. The post highlights the use of Milvus's dynamic fields (`enable_dynamic_field=True`) for flexible schema management when storing diverse content types (text, vectors, URLs, metadata). It also implicitly leverages Milvus's vector search and full-text search capabilities for querying the filtered content. The core technical contribution is the architectural pattern of using Milvus as a storage backend for an AI agent that needs to differentiate and store contextually relevant information.
  104. Introducing DeepSearcher: A Local Open Source Deep ResearchIntroduces DeepSearcher, an open-source project that enhances agentic RAG capabilities with query routing, conditional execution flow, and web crawling. It details the architecture and implementation of these features, including specific Python code for query routing and reflection prompts. The post also discusses the performance benefits of using specialized hardware for LLM inference.
  105. I Built a Deep Research with Open Source—and So Can You!This post details the construction of a research agent prototype using Milvus as the vector database. It showcases how Milvus is used to store chunks of Wikipedia articles, enabling Retrieval-Augmented Generation (RAG) queries. The implementation involves using Milvus's LangChain integration for efficient data storage and retrieval, demonstrating a practical application of the vector database in an AI agent workflow.
  106. Multimodal Semantic Search with Images and TextThis post details the construction of a multimodal semantic search application using Milvus, HuggingFace's Visualized BGE model for joint text and image embedding, and Phi-3 Vision for reranking. It showcases how to index images and text into the same vector space, perform similarity searches with combined text and image queries, and improve result relevance through LLVM-based reranking. The architecture follows a retrieve-and-rerank pattern, similar to RAG, and highlights the ability of embedding models to understand open-ended instructions in relation to query images.
  107. Getting Started with HNSWlibThis post introduces the Hierarchical Navigable Small Worlds (HNSW) algorithm and its implementation in the HNSWlib library. It explains the core concepts of HNSW, including its hierarchical layers and navigable small worlds structure, and details how HNSWlib provides efficient Approximate Nearest Neighbor (ANN) search capabilities. The post includes a step-by-step guide on setting up HNSWlib, preparing data, building an index with key parameters (`space`, `dim`, `max_elements`, `ef_construction`, `M`, `ef`), and performing nearest neighbor searches using `knn_query`. Finally, it compares HNSWlib with purpose-built vector databases like Milvus, noting HNSWlib's suitability for prototyping and medium-scale datasets versus Milvus's advantages for larger, enterprise-level requirements.
  108. Matryoshka Embeddings: Detail at Multiple ScalesThis post introduces Matryoshka embeddings, a novel embedding technique that embeds multiple scales of representation within a single vector. It explains the concept, the "funnel search" inference method, and the training objective modification. It also highlights Milvus's seamless support for these embeddings and lists popular Matryoshka embedding models.
  109. Introducing PyMilvus Integration with Embedding ModelsThis post introduces the model module integration in PyMilvus, the Python SDK for Milvus. It demonstrates how to use dense embedding models (e.g., `DefaultEmbeddingFunction` with `all-MiniLM-L6-v2`), sparse embedding models (e.g., `BM25EmbeddingFunction`), and rerankers (e.g., `JinaRerankFunction`) directly within Python applications. The integration simplifies data transformation into searchable vectors and reranking results for applications like RAG, and supports Milvus Lite for local development.
  110. Introducing Milvus Lite: Start Building a GenAI Application in SecondsIntroduces Milvus Lite, a new product offering that allows Milvus to run locally within a Python application. This is achieved by reusing core Milvus components for vector indexing and query parsing while removing elements for distributed systems. It integrates with AI development stacks like LangChain and LlamaIndex, and uses an SQLite file for persistence. The API is compatible with the standard Milvus API, with the primary difference being the `uri` parameter pointing to a local file.
  111. Unveiling Milvus 2.4: Multi-vector Search, Sparse Vector, CAGRA Index, and More!Milvus 2.4 introduces multi-vector search, enabling storage and querying of multiple vectors per entity and optimization of reranking algorithms. It adds Grouping Search for efficient large-scale grouping queries on scalar fields. Beta support for sparse vector embeddings is included, allowing hybrid search with models like SPLADEv2 and BM25, and supporting the Inner Product metric. The CAGRA index, a GPU-based graph indexing technology, is now supported for accelerated graph indexing and low-latency similarity search. Other enhancements include regex support for metadata filtering, a scalar inverted index, and a CDC tool.
  112. Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing EfficiencyIntroduced support for Apache Parquet file imports in Milvus version 2.3.4. This feature enhances data processing efficiency by leveraging Parquet's columnar storage, compression, and encoding for optimized storage and query performance. The post details how to import Parquet files directly from object storage (e.g., MinIO) and how to use the BulkWriter tool, which now defaults to Parquet output, to generate Parquet files for import. Future plans include optimizing large file imports by splitting scalar and vector columns.
  113. Milvus in 2023: An Unprecedented Vector Database Amidst Tech BuzzThis post highlights Milvus's 2023 achievements, focusing on engineering advancements. Key technical contributions include: zero downtime during rolling upgrades (refined from v2.2.3), a 3x performance improvement in production environments (especially with filtered searches and streaming insert/search), a 5% recall improvement on the Beir dataset using hybrid search (dense + sparse embeddings) and a reranking model, and 10x memory saving on large datasets through disk-based indexing using memory-mapped files (MMap) introduced in Milvus 2.3.4. Other technical developments mentioned are dynamic schema (v2.2.9), Upsert, Range Search, and Cosine metrics (v2.3), Partition Key for multi-tenancy, and enhanced memory management, coroutine handling, and CPU optimization in v2.3.4 for B2B scenarios. The post also mentions the development of VectorDBBench for benchmarking and Pyanns for sparse embedding search.
  114. Efficient Vector Similarity Search in Recommender Workflows Using Milvus with NVIDIA MerlinThis post demonstrates the integration of Milvus with NVIDIA Merlin for efficient vector similarity search in recommender workflows. It highlights Milvus's GPU acceleration capabilities, leveraging NVIDIA RAFT, to achieve significant speedups (37x to 91x) in ANN search for user and item embeddings generated by Merlin Models. The technical details cover the use of NVTabular for data preprocessing, Merlin Models for training Two-Tower deep learning models, Milvus for building GPU-accelerated indexes and performing similarity searches, and NVIDIA Triton Inference Server for the inference stage. The post also touches upon Milvus's system design, separating compute and storage for scalability, and its reliance on underlying libraries like cuDF and RAFT for GPU acceleration.
  115. How to Get the Right Vector EmbeddingsThis post provides a comprehensive introduction to vector embeddings, explaining what they are, how they are created by removing the last layer of a deep learning model, and their dimensionality. It details how to generate embeddings for images (using ResNet-50), text (using Sentence Transformers like MiniLM-L6-v2 and MPNet-Base-V2), multimodal data (using CLIP VIT), audio (using Whisper), and video (using multimodal perceiver). Code examples are provided for each data type. The post also briefly mentions the use of these embeddings with vector databases like Milvus.
  116. Revealing Milvus 2.3.2 & 2.3.3: Support for Array Data Types, Complex Delete, TiKV Integration, and MoreIntroduced support for Array data types for enhanced filtering in search. Enabled complex delete expressions for advanced data management. Integrated TiKV as an alternative to Etcd for scalable metadata storage. Added interface-level support for FP16 vector types. Improved rolling upgrade experience with reduced query impact. Implemented performance optimizations for data loading, inserts, and recall phase. Permanently deleted TimeTravel-related code and deprecated MySQL as a metadata store.
  117. Milvus Introduced MMap for Redefined Data Management and Increased Storage CapabilityIntroduced MMap (memory-mapped files) as a beta feature to redefine data management and increase storage capability. MMap maps large files directly into the system's memory space, transforming files into contiguous memory blocks and eliminating explicit read/write operations. This allows Milvus to process more data within limited memory, especially benefiting users less sensitive to performance fluctuations. Configuration involves adding `mmapDirPath` to `queryNode` in `milvus.yaml`. The feature optimizes data access based on locality, writing scalar data to disk for sequential segments and indexing variable-length data with an offsets array. Vector data uses MMap selectively while retaining adjacency lists in memory. This results in a copy-free streaming process, drastically reducing memory overhead and enabling Milvus to efficiently handle double the data volume.
  118. Comparing Vector Databases, Vector Search Libraries, and Vector Search PluginsThis post compares specialized vector databases (like Milvus) with vector search libraries (like FAISS) and vector search plugins for traditional databases. It details the technical differences in architecture, scalability, and feature sets, emphasizing why Milvus is a more optimal solution for unstructured data storage and retrieval compared to libraries and plugins, particularly for large-scale and real-time applications.
  119. Ingesting Chaos: The MLOps Behind Handling Unstructured Data Reliably at Scale for RAGThis post details the MLOps behind handling unstructured data for RAG, including data ingestion pipelines, chunking strategies, metadata orchestration, embedding model selection, and automated evaluation systems for tuning retrieval parameters. It emphasizes the use of tools like VectorFlow and Milvus for production RAG systems.
  120. Unveiling Milvus 2.3: A Milestone Release Offering Support for GPU, Arm64, CDC, and Many Other Highly Anticipated FeaturesMilvus 2.3 introduces GPU indexing using the RAFT algorithm via Knowhere, leading to 3-10x faster QPS. It adds Arm64 support for Docker images. The upsert operation is implemented as a delete-then-insert combination. Range search allows specifying distance bounds for queries. ScaNN index is now supported for faster queries. The Growing Index automatically indexes streaming data. An iterator interface in Pymilvus enables batch retrieval of over 16,384 entities. MMap support increases single-machine data capacity. CDC is introduced for data synchronization and backup. Performance improvements include optimized HNSW filtering, full multi-core CPU utilization, a refactored stateless QueryNode, and a new message queue system using NATS and Bookkeeper. The load balancer is optimized for faster fault detection and adjustment.
  121. Conversational Memory in LangChainThis post details the integration of Milvus as a vector store for LangChain's conversational memory. It outlines the setup process involving LangChain libraries, OpenAI embeddings, and Milvus Lite. The core contribution is demonstrating how to use `VectorStoreRetrieverMemory` with Milvus to store and retrieve conversational context, enabling LLMs to maintain state and provide more coherent interactions. It provides code examples for initializing Milvus Lite, creating a retriever, saving conversation context, and querying the memory within a `ConversationChain`.
  122. Milvus 2.2.9: A Highly Anticipated Release with Optimal User ExperienceIntroduced support for JSON data types, allowing storage and querying of JSON alongside vector metadata. Enabled dynamic schema functionality via SDK, automatically filling dynamic fields into a hidden JSON field. Implemented Partition Key feature, allowing user-specific columns as primary keys for partitioning, removing partition limits and the need for explicit partition load/release APIs. Added support for Alibaba Cloud OSS.
  123. What's new in Milvus 2.1 - Towards simplicity and speedMilvus 2.1 introduces significant performance enhancements including a new routing protocol reducing retrieval latency to 5ms, a new cost evaluation model and concurrency scheduler for a 3.2x performance boost with small-nq queries, and in-memory replicas for improved scalability and availability. Data loading is accelerated by Zstandard compression and goroutine pools. String data types (VARCHAR) are now supported as scalar data, usable as primary keys and for attribute filtering, with MARISA-Tries used for scalar inverted indexes to improve filtering speed. Kafka is added as an alternative message queue to Pulsar. A production-ready Java SDK is released, and observability is improved with new monitoring metrics and optimized log keeping. Embedded Milvus is introduced for simplified local development.
  124. Building an Intelligent Wardrobe Customization System Powered by Milvus Vector DatabaseThis post details the use of Milvus for building an intelligent wardrobe customization system, leveraging vector similarity search for complex data modeling and filtering of wardrobe components. It describes the system architecture involving Elasticsearch for coarse filtering and Milvus for fine-grained similarity search and ranking, detailing the offline and online data flows for preparing and processing wardrobe data into vectors.
  125. How Short-video Platform Likee Removes Duplicate Videos with MilvusThis post details how Likee, a short-video platform, uses Milvus to remove duplicate videos. The process involves cutting videos into frames, converting each frame into a feature vector using deep learning models, and then searching for the top K most similar vectors in a database of 700 million vectors stored in Milvus. The system architecture involves Kafka for real-time ingestion, deep learning models for feature extraction, Milvus for similarity search, Ceph for storing indexed feature vectors, and TiDB or Pika for storing corresponding video IDs. The similarity search procedure involves batch searching for top 100 similar vectors, comparing video IDs to identify duplicates, retrieving feature vectors of remaining videos, and finally calculating similarity scores to determine the final result.
  126. Scalable and Blazing Fast Similarity Search with Milvus Vector DatabaseThis post introduces the fundamental concepts of vector databases and similarity search, explaining how unstructured data is represented as vectors (embeddings) and the challenges of storing, representing, and querying this data. It details the necessity of vector databases for handling massive amounts of vectors efficiently, outlining key requirements such as scalability, reliability, and speed. The post then introduces Milvus as an open-source vector database, highlighting its role in accelerating MLOps and its capabilities in storing, indexing, and querying embedding vectors, including support for attribute filtering using a bitmask mechanism.
  127. Zhentu - the Photo Fraud Detector Based on MilvusThis post details the application of Milvus for a photo fraud detection system (Zhentu) by BestPay. It describes the system architecture involving image pre-processing, feature vector extraction using deep learning, normalization, and vector search with Milvus. The deployment section outlines using Milvus cluster on Kubernetes with specific index parameters (IVF_PQ, nlist=2048) and search parameters (IP, nprobe=32) for optimizing performance. Real-world performance metrics are provided, showing sub-second query times for tens of millions of vectors.
  128. Raft or not? The Best Solution to Data Consistency in Cloud-native DatabasesThis post analyzes the limitations of consensus-based replication (Raft, Paxos) in cloud-native databases, particularly concerning availability, complexity, and performance. It proposes an alternative log replication strategy for cloud-native and distributed databases based on two principles: 1) Replication as a service, decoupling synchronization from storage, and 2) The 'Russian doll principle', leveraging external services for logging (e.g., Kafka/Kinesis) and storage (e.g., S3), and using coordination services (e.g., etcd) for leader election. The post also touches upon Milvus's tunable consistency features, such as follower reads, as a means to improve cluster throughput and reduce leader load.
  129. What Powers Similarity Search in Milvus Vector Database?This post introduces Knowhere, Milvus's core vector computation engine. It details Knowhere's concept, its role within the Milvus architecture, and its advantages over Faiss, including support for BitsetView, more similarity metrics for binary vectors, and AVX512 instruction sets with automatic SIMD instruction selection. The post also breaks down the Knowhere code structure, explaining base classes and various index types (IDMAP, IVF, Annoy, HNSW), and provides a guide on how to add new indexes to Knowhere.
  130. How Does the Database Understand and Execute Your Query?This post details the internal workings of Milvus's query execution engine. It explains how query expressions with scalar filtering are parsed using EBNF syntax, how an Abstract Syntax Tree (AST) is generated using ANTLR and the Expr package, and how the query is executed through a visitor design pattern on protobuf structures, resulting in a bitmask output. It also provides an example of a complex film query and its step-by-step processing.
  131. Data Insertion and Data Persistence in a Vector DatabaseThis post details the data insertion workflow in Milvus, explaining the roles of proxy, data coord, data node, and root coord. It elaborates on data organization (collection, partition, shard/channel, segment) and the different states of segments (growing, sealed, flushed). It also describes data persistence through binlog files and the mechanisms for segment allocation and channel assignment (consistent hashing, load balancing). The Time Tick mechanism for timestamping and data consumption is also explained.
  132. Building a Vector Database for Scalable Similarity SearchThis post details the thinking process and design principles behind building Milvus, an open-source, cloud-native vector database for scalable similarity search. It explains the Milvus architecture in detail, covering the need for a basic software stack for unstructured data, the conversion of unstructured data to vectors, and the role of similarity metrics. It differentiates Milvus as a vector database from a vector search engine by highlighting features like sharding, partitioning, replication, disaster recovery, load balancing, and query parsing. The cloud-native approach is discussed, including the separation of compute and storage ('shared something' architecture) and the use of Kubernetes for microservices. The design principles of Milvus 2.0 are elaborated: 'Log as data' where logs record all changes and worker nodes consume them; 'Duality of table and log' where logs can be converted to tables and aggregated into segments; and 'Log persistency' using a publish-subscribe system (log broker) for decoupling and statelessness. The deployment modes (standalone and cluster) are described, along with a high-level overview of the Milvus architecture's four layers: access, coordination, distributed execution, and storage.
  133. How to Compact Data in Milvus?This post details the concept and implementation of data compaction in Milvus 2.0. It explains that compaction merges small segments into large ones and cleans logically deleted data to save storage space and improve query efficiency. It covers configuration parameters (`dataCoord.enableCompaction`, `common.retentionDuration`), manual and automatic triggering mechanisms, and the specific processes for binlog compaction (based on delta binlog row count or size) and segment compaction (based on the number of small sealed segments).
  134. How Bitset Enables the Versatility of Vector Similarity SearchThis post details the internal mechanism of bitsets in Milvus, explaining how they are used to support Time Travel, attribute filtering, and data deletion. It provides concrete examples of bitset generation and manipulation based on DML events and query timestamps, illustrating the logical operations involved in determining which entities are included in search or query results.
  135. How Milvus Deletes Streaming Data in a Distributed ClusterThis post details the implementation of the DELETE function in Milvus 2.0, enabling logical deletion of entities. It explains how data nodes handle INSERT and DELETE messages, including the use of bloom filters and Delta binlogs. It also describes how query nodes process these messages, distinguishing between growing and sealed segments, and how deleted entities are omitted during search operations using bitsets. The post also touches upon the impact of consistency levels on the visibility of deleted data.
  136. The Story of SmartNews - from a Milvus User to an Active ContributorIntroduced the concept and implementation of collection aliases as a mechanism for hot reloading data in Milvus. This allows for dynamic updates of large datasets by swapping aliases to point to new collections, avoiding the need for data deletion/insertion or client code changes. The implementation was contributed by SmartNews and officially released with Milvus 2.0.0-RC7.
  137. Evolution of Milvus Cloud-scalable Vector DatabaseThis post details the design thinking behind Milvus 2.0's cloud-scalable cluster architecture. It contrasts with traditional shared-everything clusters, highlighting the challenges of global locking and buffer cross-invalidation in cloud environments. Milvus 2.0 adopts an eventual consistency model and asynchronous processes, utilizing cloud messaging queues (Pulsar/Kafka) for log storage and data nodes to bridge local storage with cloud shared storage (S3). The architecture aims for high scalability and availability by moving away from centralized components and embracing a more distributed, event-driven approach. Data is managed in collections with physical segments, and partitioning is discussed as a strategy for performance.
  138. Milvus in IP Protection：Building a Trademark Similarity Search System with MilvusThis post details the construction of a trademark similarity search system using Milvus. It outlines a four-step process: preparing a logo dataset, training an image feature extraction model (specifically VGG16), converting logos into vectors, and storing/searching these vectors in Milvus. The post emphasizes the use of Milvus's vector similarity search capabilities and suggests organizing data into collections based on the International Classification of Goods and Services for improved search performance.
  139. Accelerating Candidate Generation in Recommender Systems Using Milvus paired with PaddlePaddleThis post details a product recommender system project that uses Milvus for candidate generation. It explains the basic workflow of a recommender system (candidate generation and ranking), introduces the MIND algorithm for multi-interest user representation, PaddleRec for model training and deployment, and Milvus for storing and searching product item vectors. The implementation details cover data processing, model training using MIND on PaddleRec, and generating product item candidates by storing item vectors in Milvus and performing vector similarity search with user vectors.
  140. Frustrated with New Data? Our Vector Database can HelpThis post introduces Milvus as an AI-oriented general-purpose vector database system designed to handle unstructured data. It discusses the challenges of traditional data analytics with unstructured data and the emergence of the vector data layer through embedding technology. The post details the major attributes of a vector database, including support for high-efficiency vector operators, vector indexing, consistent user experience, hybrid search, and cloud-native architecture. It then presents the Milvus 2.0 architecture, emphasizing 'log as data', unified batch and stream processing, stateless components, and microservices, with detailed explanations of data persistence through metadata, log, and object storage, and the role of the log broker in ensuring data reliability and decoupling components.
  141. Building a Video Analysis System with Milvus Vector DatabaseThis post details the construction of a video analysis system leveraging Milvus. It outlines the process of extracting frames from videos, performing object detection using YOLOv3, extracting feature vectors from detected objects with ResNet-50, and finally utilizing Milvus for vector similarity search to analyze video content. Code snippets demonstrate frame extraction with OpenCV, object detection, feature vector extraction, and Milvus search operations.
  142. Combine AI Models for Image Search using ONNX and MilvusThis post details a process for combining multiple AI models (VGG16 and ResNet50) for image search. It demonstrates converting Keras models to ONNX format, extracting normalized feature vectors using ONNX Runtime, storing these vectors in separate Milvus collections, and performing similarity searches using Euclidean distance. It also highlights the use of Redis to map vector IDs back to image paths, and notes the limitations of Milvus 1.1 regarding scalar filtering.
  143. DiskANN, A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale DatasetThis post introduces and explains DiskANN, a disk-based ANNS solution inspired by the NeurIPS 2019 paper. It details the motivations for such a solution, highlighting the limitations of memory-resident graph-based indexing for large datasets. The post describes the Vamana algorithm, a novel graph-based approach with a flexible trimming strategy, and how DiskANN adapts it for SSD-resident indexing. Key optimizations like caching hotspots and beam search are explained to reduce disk I/O. The post also includes experimental comparisons of Vamana with NSG and HNSW, and outlines the experimental setup for DiskANN.
  144. DNA Sequence Classification based on MilvusThis post details the application of Milvus for DNA sequence classification. It explains the k-mer preprocessing technique to convert DNA sequences into manageable text-like data, followed by vectorization using NLP models like CountVectorizer. The post demonstrates how Milvus's similarity search capabilities, based on ANN, can be used to classify gene families and determine genetic similarity between organisms. It includes code snippets for inserting vectors and performing searches, and presents performance comparisons showing Milvus outperforming MySQL and other ML classifiers in accuracy for DNA sequence classification.
  145. Paper Reading｜HM-ANN When ANNS Meets Heterogeneous MemoryThis post analyzes the HM-ANN algorithm, a novel approach for Approximate Nearest Neighbor (ANN) search that addresses memory heterogeneity by optimizing index placement across DRAM and Persistent Memory (PMem). It details HM-ANN's graph construction (top-down insertion, bottom-up promotion with high-degree node selection) and search algorithms (fast memory search, parallel layer-0 search with prefetching). The evaluation compares HM-ANN against other ANN algorithms like HNSW and Disk-ANN on billion-scale datasets, highlighting its performance benefits in terms of latency and recall, especially when datasets exceed DRAM capacity. It also discusses the effectiveness of high-degree promotion and the performance gains from memory management techniques like data prefetching.
  146. Milvus 2.0 Redefining Vector DatabaseMilvus 2.0 is a refactored cloud-native vector database built on principles of cloud-native first, logs as data, and unified batch and stream processing. It features a microservice architecture with storage and computing separation, read/write separation, and incremental/historical data separation. The system architecture includes an access layer, coordinator service (root coord, data coord, query coord, index coord), worker nodes (data nodes, query nodes, index nodes), and storage (etcd for meta, Pulsar for log broker, MinIO/S3 for object storage). Key features introduced include always online capabilities through "fail cheap, fail small, fail often" strategies, hybrid search between scalar and vector data, tunable consistency, time travel for data rollback, and an ORM Python SDK. Supporting tools like Milvus Insight (GUI) and out-of-box deployment via helm/docker-compose are also highlighted.
  147. Building a Personalized Product Recommender System with Vipshop and MilvusThis post details Vipshop's implementation of a personalized product recommender system using Milvus. It describes a read/write separation architecture, the use of Redis for managing collection aliases to enable seamless data updates (effectively near real-time updates by switching entire data collections), batch writing data to multiple partitions, index building and pre-loading for performance, and a data recall process that achieves an average latency of 30ms. It also highlights lessons learned regarding connection pooling for the Java client, index warm-up, and parameter tuning (nlist, nprobe).
  148. Audio Retrieval Based on MilvusThis post details the application of Milvus for audio retrieval. It explains how audio data is processed by extracting feature vectors using pre-trained deep learning models (PANNs) and then inserted into Milvus for vector similarity search. The system architecture is described, encompassing data insertion (generating embeddings, inserting into Milvus, correlating IDs with metadata in MySQL) and audio search (extracting query embeddings, searching Milvus, retrieving audio names from MySQL). It highlights Milvus's role in managing and querying these feature vectors for unstructured data retrieval.
  149. MilMil A Milvus-powered FAQ Chatbot that Answers Questions About MilvusThis post details the architecture and implementation of MilMil, a Milvus-powered FAQ chatbot. It explains how Milvus is used for vector similarity retrieval to answer user questions. The system utilizes the `sentence-transformers/paraphrase-mpnet-base-v2` model for generating vector representations of FAQs, which are then stored in Milvus. PostgreSQL is used to store question-answer pairs linked by vector IDs. When a user submits a question, it's converted to a vector, and Milvus retrieves the most semantically similar vectors. The corresponding questions and answers are then presented to the user. The post also provides a link to the relevant code in the Milvus bootcamp.
  150. Building a Wardrobe and Outfit Planning App with MilvusThis post details the implementation of an image similarity search system for the Stylepedia app using Milvus. It covers the system overview, breaking down the implementation into garment detection (YOLOv5), feature extraction (EfficientNet with deep metric learning), vector similarity search (Milvus with HNSW index and partitioning), and post-processing. The system vectorizes images into 512-dimensional feature vectors and uses Milvus for near real-time search on millions of vectors, enabling features like user uploads, outfit suggestions, and fashion photograph recommendations.
  151. Building a Milvus Cluster Based on JuiceFSThis post details the integration of Milvus with JuiceFS, a cloud-native distributed POSIX file system. It explains how to configure Milvus to use JuiceFS for shared file storage, including setting up MySQL for metadata, creating a JuiceFS file system on object storage (e.g., Azure Blob Storage) and Redis for metadata, and configuring Milvus nodes (read/write roles) and Mishards. Performance benchmarks are presented, highlighting JuiceFS's advantages over EFS in IOPS and throughput, and demonstrating near-instantaneous data loading from disk to memory.
  152. Building an Intelligent News Recommendation System Inside Sohu News AppThis post details how Sohu News integrated Milvus into their news recommendation system. It explains the use of semantic vector search with BERT-as-service for encoding news articles and user profiles, the workflow for generating user semantic vectors daily, and the insertion of these vectors into Milvus. A key technical contribution is the optimization strategy using MinHash and Jaccard distance to reduce semantic vector extraction time from days to hours. Additionally, it describes a method for resolving short text classification errors by querying Milvus with short text vectors to find similar long text articles and using a voting mechanism for reclassification. The post highlights Milvus's role in improving real-time performance and the efficiency of text classification.
  153. Accelerating AI in Finance with Milvus, an Open-Source Vector DatabaseThis post details how Milvus, an open-source vector database, can be applied in the financial services industry to accelerate AI initiatives. It outlines specific use cases including enhancing customer experience with banking chatbots, boosting sales with recommender systems, and analyzing financial data through semantic text mining. The post emphasizes Milvus's capability to manage and search massive vector datasets, enabling similarity searches for AI applications.
  154. What's Inside Milvus 1.0?This post introduces Milvus v1.0, detailing its core features and optimizations. It covers CPU/GPU distributions, vector entity structure with integer IDs, partitioning for search acceleration, and enhancements to index algorithms (IVF, FLAT, IVF_SQ8H). Performance optimizations for search (nq < threads, request combining) and index building are highlighted. It also mentions distance metrics, logging configuration, the Mishards distributed solution, Prometheus integration for monitoring, and metadata management (SQLite/MySQL).
  155. Making with Milvus Detecting Android Viruses in Real Time for Trend MicroThis post details Trend Micro's implementation of Milvus for real-time Android virus detection. It describes their system architecture, which involves converting APK Thash values into binary vectors and storing them in Milvus for similarity search. The post highlights the use of Hamming distance and specific index types (FLAT, IVF_FLAT) for binary vectors. It also discusses the integration with MySQL and Redis for storing Sha256 values and caching, respectively. The deployment utilizes Mishards for sharding and Kubernetes for high availability. Monitoring with Prometheus and Grafana is also covered.
  156. Build Semantic Search at SpeedThis post details how Milvus, leveraging FAISS, can be used as a vector search engine to achieve high-performance semantic search, demonstrating its capability to handle millions of vectors and thousands of requests per second with low latency. It highlights the integration of Milvus with other systems like Solr for a complete semantic search workflow and discusses the performance implications of different index types (FLAT vs. HNSW) and scale testing results.
  157. How to Make 4 Popular AI Applications with MilvusThis post demonstrates the practical application of Milvus in building four distinct AI applications: NLP chatbots, reverse image search, audio search, and video object detection. It outlines the specific ML models (BERT, VGG, PANNs, YOLOv3, ResNet50) used in conjunction with Milvus for each use case, detailing the data processing steps (vectorization and storage) and search mechanisms. For NLP, it explains how BERT converts questions to vectors, and Milvus retrieves similar questions, with answers stored in PostgreSQL. For image search, VGG converts images to vectors, Milvus finds similar vectors, and image paths are stored in CacheDB. For audio search, PANNs convert audio to vectors, Milvus finds similar vectors using IP distance, and audio paths are stored in PostgreSQL. For video object detection, ResNet50 converts object images to vectors, Milvus finds similar objects, and image paths are stored in MySQL, with OpenCV and YOLOv3 used for video processing.
  158. Operationalize AI at Scale with Software 2.0, MLOps, and MilvusThis post introduces MLOps (Model Operations) as a systemic approach to AI model lifecycle management, contrasting it with traditional DevOps. It explains the core functionalities of MLOps, including CI/CD, Model Development Environments (MDE), champion-challenger testing, model versioning, and model store/rollback. The article positions Milvus as a vector data management platform that supports the transition to Software 2.0 and operationalizes AI at scale by facilitating the storage, querying, updating, and maintenance of massive vector datasets, integrating with index libraries, and offering APIs for popular frameworks.
  159. Making With Milvus AI-Infused Proptech for Personalized Real Estate SearchThis post details how Beike used Milvus to build an AI-infused proptech platform for personalized real estate search. It explains the concept of vector similarity search, how Milvus manages vector data, and Beike's specific implementation. Beike converts property listings into feature vectors, indexes them in Milvus, and uses similarity search (cosine distance) to recommend similar homes based on user preferences and listing characteristics. The system achieves an average query time of 113ms on 3 million vectors and uses an A/B table switching mechanism for data updates.
  160. Extracting Event Highlights Using iYUNDONG Sports AppThis post details iYUNDONG's implementation of an intelligent image retrieval system using Milvus to power their sports app. It highlights the challenges of handling massive datasets, achieving instant search availability, and efficient feature extraction. Milvus was chosen for its ability to perform fast vector retrieval on ultra-large datasets and mass storage at lower costs, enabling millisecond-level queries on over 60 million images with an accuracy rate above 92%. The system architecture involves converting facial features into 128-dimensional vectors and storing them in Milvus, with additional metadata stored in a third-party database, linked via the Leaf algorithm for distributed ID generation.
  161. Milvus 1.0 The World's Most Popular Open-Source Vector Database Just Got BetterMilvus v1.0 release introduces support for a wider range of similarity metrics, improved integration with state-of-the-art ANNs algorithms, and enhanced scale-out capabilities through the Mishards sharding proxy. It also adds support for various AI processors like X86, Nvidia GPU, and Xilinx FPGA. The release signifies a commitment to long-term support for v1.0 and a focus on developing toolchain enhancements and fostering AI ecosystem integration. Future development will utilize Golang for the Milvus engine and C++ for ANNS algorithms, with the distributed version aiming to utilize existing cloud components.
  162. Building Personalized Recommender Systems with Milvus and PaddlePaddleThis post details the integration of Milvus with PaddlePaddle for building personalized recommender systems. It outlines the data preparation using the MovieLens Million Dataset, the Fusion Recommendation Model implemented in PaddlePaddle, and the system overview where movie feature vectors are stored in Milvus and user features are used for similarity search. It provides steps for model training, data preprocessing, and implementing the recommender system with Milvus, demonstrating how to insert movie feature vectors and perform similarity retrieval using user features.
  163. How we used semantic search to make our search 10x smarterThis post details Tokopedia's implementation of semantic search using Milvus. It covers their transition from keyword-based search with Elasticsearch to vector representation and similarity search. The decision to use Milvus was based on its ease of use and comprehensive index support compared to FAISS. The post also describes the deployment of a Milvus cluster with Mishards for high availability, and explains the fundamental concept of vector indexing for accelerating similarity search.
  164. Milvus Was Built for Massive-Scale (Think Trillion) Vector Similarity SearchThis post details Milvus's architecture for massive-scale vector similarity search, focusing on its use of LSM trees for efficient dynamic data management, optimized data handling with WAL and buffer flushing, segment merging to reduce fragmentation, and the acceleration of similarity search through ANN indexing. It also mentions the use of OLTP systems (SQLite/MySQL) for metadata management.
  165. Milvus Is an Open-Source Scalable Vector DatabaseThis post introduces Milvus as an open-source data management platform for massive-scale vector data and MLOps. It highlights the use of AI to convert unstructured data into vectors and the necessity of ANN algorithms for efficient similarity search. Milvus integrates libraries like Faiss, NMSLIB, and Annoy to provide a unified platform for storing, processing, and analyzing large vector datasets. It supports CPU/GPU computing, distributed architecture for scalability, and compatibility with various AI models, programming languages, and processor types.
  166. Vector Similarity Search Hides in Plain ViewThis post provides a high-level overview of vector similarity search, explaining what it is, its applications (e-commerce, security, recommendation engines, chatbots, image/video search, chemical similarity search), and introduces Milvus as an open-source vector data management platform. It discusses the conversion of unstructured data to feature vectors using neural networks and the use of Approximate Nearest Neighbor (ANN) algorithms to accelerate similarity search.
  167. Building a Graph-based Recommendation System with Milvus, PinSage, DGL, and MovieLens DatasetsThis post details the integration of Milvus as a vector database within a graph-based recommendation system. It outlines the process of using PinSage and DGL to generate movie embeddings from the MovieLens dataset, loading these embeddings into Milvus for storage and indexing, and then performing vector similarity searches to retrieve similar movie information. The post also touches upon data preprocessing and the general workflow of building such a system.
  168. Thanks to Milvus, Anyone Can Build a Vector Database for 1+ Billion ImagesThis post introduces Milvus as a vector data management platform capable of building large-scale unstructured data analytics systems, specifically highlighting its ability to handle 1+ billion images for reverse image search. It details how AI enables unstructured data analytics by converting data into feature vectors, discusses the limitations of existing approaches (flexibility, dynamic data handling, advanced query processing, heterogeneous computing), and positions Milvus as a solution addressing these challenges. Milvus supports multiple SDKs, index types, and provides optimizations for CPUs and GPUs, achieving high recall rates and fast query times on massive datasets.
  169. ArtLens AI Share Your ViewThis post details the implementation of the ArtLens AI: Share Your View tool by the Cleveland Museum of Art's Digital Innovation team, in collaboration with Design IO and Jeff Schuler. The tool uses AI and the open-source vector similarity engine Milvus to match user photos to artworks from the museum's collection. It highlights the use of reverse image search to recognize shapes, patterns, and objects in user photos and find corresponding art pieces. The post also mentions the integration of AI used in the museum's on-site ArtLens Exhibition.
  170. Item-based Collaborative Filtering for Music Recommender SystemThis post details the implementation of an item-based collaborative filtering (I2I CF) recommender system for music, leveraging Milvus as the embedding similarity search engine. It describes the process of extracting song feature embeddings using CNNs from Mel-frequency cepstrum (MFC) and deploying Milvus in a highly available cluster configuration with Mishards on Kubernetes for production. Additionally, Milvus is used as a preliminary step for duplicate song filtering by searching for similar audio fingerprints.
  171. 4 Steps to Building a Video Search SystemThis post details the architecture and implementation of a video search system using Milvus 0.7.1. It outlines the process of extracting key frames from videos, generating embeddings using the VGG model, and storing these embeddings in Milvus. The system also utilizes Minio for video storage and Redis for mapping correlations between videos and vectors. Deployment is managed via Docker and docker-compose, with a specific script for importing video data.
  172. How Milvus Realizes the Delete FunctionIntroduced a soft deletion mechanism for flushed data by saving deleted IDs in a separate DEL file, which are filtered during read operations. Implemented segments as the basic unit for data operations, containing raw files, index files, and a DEL file. Utilized counting bloom filters per segment to optimize the identification of segments containing deleted IDs.
  173. The Journey to Optimizing Billion-scale Image Search (2/2)This post details the technical implementation of a search-by-image system using CNNs (VGG16) for feature extraction and Milvus as the vector search engine. It covers feature extraction using Keras/TensorFlow, image preprocessing (normalization, bytes conversion, black border removal), Milvus requirements (CPU instruction set), capacity planning for billion-scale datasets (calculating storage for 512-dimensional float32 vectors), system configuration, database design considerations (collections, partitions, metadata management with SQLite/MySQL, limitations on number of collections/partitions), structured data and vector mapping via IDs, index selection, and processing search results (ID + distance, filtering -1 IDs, pagination).
  174. The Journey to Optimizing Billion-scale Image Search (1/2)This post details the first-generation image search system built by Yupoo Picture Manager, which used the pHash algorithm for feature extraction and Elasticsearch for similarity calculation based on Hamming distance. It explains the limitations of pHash, particularly its inability to handle image integrity destruction, and sets the stage for a second-generation system that would likely employ more advanced vector embedding techniques for billion-scale image search.
  175. Building an AI-Powered Writing Assistant for WPS OfficeThis post details the technical implementation of an AI-powered writing assistant for WPS Office. It describes the process of making sense of unstructured textual data, using TFIDF for feature extraction, employing a bi-directional LSTM-CNNs-CRF deep learning model for named entity recognition, and creating sentence embeddings with Infersent. Milvus is used as the similarity search engine to store and query these embeddings, utilizing the IVF_FLAT index and partitioning for performance optimization. The system runs on Kubernetes clusters and uses MySQL for metadata management.
  176. Making with Milvus AI-Powered News Recommendation Inside Xiaomi's Mobile BrowserThis post details Xiaomi's implementation of an AI-powered news recommendation engine using Milvus as the core data management platform. It explains the two-stage recommendation process (retrieval and sorting) and highlights the integration of Milvus with the BERT language model. Specifically, it describes the vectorization process using SimBert (an optimized BERT model), the role of ANN algorithms for similarity search within Milvus, and the ID mapping for retrieving article metadata. The post also touches upon data update strategies (full and incremental) and the real-time incorporation of new data into Milvus for continuous recommendation updates.
  177. Building an Intelligent QA System with NLP and MilvusThis post details the architecture and implementation of an intelligent QA system using BERT for semantic understanding and Milvus for vector search. It covers data preparation from the insurance industry corpus, generating feature vectors using a pre-trained BERT model, importing these vectors into Milvus with associated IDs, and storing the question IDs and their corresponding answers in PostgreSQL. The retrieval process involves converting user questions to feature vectors, performing similarity search in Milvus using cosine distance, and fetching the answer from PostgreSQL based on the retrieved ID. A similarity threshold is suggested for handling cases where no closely matching question is found.
  178. How Milvus Implements Dynamic Data Update And QueryThis post details Milvus's internal mechanism for handling dynamic data updates and near real-time queries. It introduces the MemTable and MemTableFile architecture for managing data in memory, dividing MemTableFiles into mutable and immutable states. Data is inserted into a mutable MemTableFile, which transitions to an immutable state and is then serialized to disk by a background thread. This process ensures data is available for searching shortly after insertion, with a maximum latency of one second. The post also touches upon merging serialized files and asynchronous index building.
  179. Mishards — Distributed Vector Search in MilvusIntroduces Mishards, a distributed solution for Milvus enabling horizontal scaling for massive datasets (10B+ vectors). Details the Mishards architecture including service discovery (ZooKeeper, etcd, Consul), load balancing (Nginx, HAProxy), Mishards nodes (stateless, scalable), write-only Milvus nodes (single, HA required), read-only Milvus nodes (stateful, scalable), shared storage, and metadata service (MySQL). Explains how Mishards breaks down requests, routes them to read-only nodes, and merges results. Discusses read-only node scalability through sharding and horizontal scaling. Highlights the integration with OpenTracing for service invocation tracing and Prometheus/Grafana for monitoring.
  180. How to Select Index Parameters for IVF IndexThis post provides best practices for selecting index parameters for the IVF index type in Milvus. It details the impact of `index_file_size` on search performance and memory loading, and explains the roles of `nlist` (number of clusters) and `nprobe` (number of clusters to search) in balancing search performance, precision, and computation load. It includes experimental results and recommendations for setting these parameters.
  181. Accelerating New Drug DiscoveryThis post details the application of Milvus for chemical fingerprint similarity search using RDKit. It explains how chemical fingerprints are generated using RDKit's Morgan fingerprints and then imported into Milvus for substructure, similarity, and exact search. The post also clarifies the relationship between Tanimoto distance and Jaccard distance in the context of chemical fingerprint computation within Milvus, and presents a demo of searching over 90 million chemical fingerprints.
  182. Milvus Metadata Management (2)This post details the fields within the `Tables` and `TableFiles` metadata tables in Milvus versions 0.5.0 and 0.6.0. It explains the significance of fields like `index_file_size`, `engine_type`, `nlist`, `metric_type`, `state`, `created_on`, `owner_table`, `partition_tag`, and `version`. It also describes fields in `TableFiles` such as `file_type`, `file_size`, and the number of vector rows, and clarifies the relationship between data files and their corresponding index files. The post also highlights the evolution of the data file cleanup mechanism in v0.6.0, moving from a fixed interval hard-delete to a reference counting approach for improved reliability and reduced disk usage.
  183. Milvus Metadata Management (2) Fields in the Metadata TableThis post details the fields within Milvus's metadata tables (`Tables` and `TableFiles`) for versions 0.5.0 and 0.6.0. It explains the purpose and data types of fields related to table configuration (e.g., dimension, index type, metric type), state, and file management (e.g., file size, row count, file type). It also introduces new fields added in 0.6.0 for partition management, such as `owner_table` and `partition_tag`.
  184. Milvus Metadata Management (1)This post details how to view Milvus metadata by configuring and querying either SQLite or MySQL as the metadata backend. It provides specific commands for setting up SQLite and MySQL clients and for querying the `Tables` and `TableFiles` tables.
  185. Milvus Metadata Management (1) How to View MetadataThis post details how to view Milvus metadata by connecting to either SQLite or MySQL databases. It provides specific command-line instructions for installing clients and querying the `Tables` and `TableFiles` tables, demonstrating how to inspect the internal structure of Milvus's data management.
  186. Accelerating Similarity Search on Really Big Data with Vector IndexingThis post details the role of vector indexing in accelerating similarity search, focusing on different Vector Inverted File (IVF) index types: FLAT, IVF_FLAT, IVF_SQ8, and IVF_SQ8H. It provides performance test results for query time and recall rates for each index type in Milvus, offering guidance on selecting the best index for various scenarios based on dataset size, accuracy requirements, and resource availability. It also introduces the concept of scalar quantization in IVF_SQ8 for reduced resource consumption.
  187. Managing Data in Massive-Scale Vector Search EngineThis post details Milvus's data management strategies for massive-scale vector search. It explains vector insertion using a mutable buffer that flushes to disk regularly, similar to Elasticsearch. It describes the creation and merging of Raw Data Files to manage fragmentation and the process of building Index Files for efficient search, contrasting them with raw data. The post also covers Meta Data management using SQLite/MySQL to track file statuses and operations, and introduces the Query Scheduler for optimizing hardware resource utilization during searches, including 'cold' vs. 'warm' queries and LRU for data replacement. Finally, it touches upon the Result Reducer for combining search results.