AI Data Storage Engine

7/30/2026

Introduced Milvus Snapshots, a feature that creates point-in-time, read-only views of collections by referencing existing data files instead of copying them. This enables lightweight snapshot creation, fast restores by reusing index files, cross-cluster data movement through metadata URIs, and stable views for external consumers and batch jobs. The implementation leverages the write-once nature of Milvus storage files and records file references and metadata to represent a collection's state at a specific moment. This significantly reduces the cost and time for tasks like rollback, testing, and batch processing.

7/27/2026

Milvus 3.0 introduces a lake-native path for indexing and serving vector data directly from object storage using open table formats (Parquet, Lance, Iceberg, Vortex) via External Collections. It enhances the retrieval engine with server-side sorting, aggregation, faceted search, StructArray, and an improved sparse index. Loon (Storage v3) is a new manifest-based columnar storage engine for S3-compatible object storage designed to reduce read amplification for point reads. Snapshots provide point-in-time, read-only views for offline jobs. A Spark DataSource V2 connector is added for batch workflow integration. Online schema evolution (adding/dropping columns) is now supported without downtime. New data model features include nullable vectors, TEXT LOB, TTL, MinHash, and Woodpecker.

7/22/2026

This post describes a specific, non-standard data recovery procedure for Milvus instances suffering from etcd corruption without a backup. It details the steps involved: starting a clean Milvus instance, recreating the collection schema, copying old MinIO binlogs to a recovery prefix, using the Java SDK's bulkInsert API in backup mode to import candidate segments, and validating each segment before inclusion in the final recovered collection. It also explains why MinIO alone cannot rebuild a collection and the importance of validating segments to avoid duplicates.

7/2/2026

Introduces Open Tag, an open-source implementation of the Claude Tag pattern, leveraging the MFS (Multi-source File-like Search) system. Details the architecture of Open Tag, highlighting MFS as the core component for agent memory and tools. Explains MFS's design principles: a unified URI interface across heterogeneous sources and a retrieval loop combining search and browse. Demonstrates Open Tag's functionality with examples of local installation, agent-guided configuration, and a real-world PR review. Discusses the engineering problem of enabling agents to search and retrieve context from multiple isolated systems efficiently.

7/1/2026

Introduces and details the 'Force Merge Compaction' feature for Milvus. Explains its mechanism for consolidating small segments into larger ones, contrasting it with standard compaction. Provides experimental results demonstrating significant search QPS improvements (76%-87%) and latency reduction on a 1 million vector dataset. Outlines the conditions under which Force Merge is beneficial and when to avoid it, emphasizing its role as a maintenance action for static, read-heavy collections.

6/18/2026

This post extends the discussion on AI agents by introducing a four-level autonomy ladder (L1-L4) to frame their development and impact. It specifically elaborates on L2 agents' memory capabilities and the role of systems like Milvus in enabling this by providing a searchable retrieval layer for agent memory. The post also emphasizes the critical need for L4 agents that focus on improving human judgment, arguing that without this, the standards agents operate under will stagnate. It frames the evolution of AI agents not just as task execution but as a loop where human judgment defines standards, agents execute, and feedback improves human learning, which in turn refines the standards.

6/10/2026

Introduces Loon, a novel storage engine built to handle the dynamic nature of AI data. It details the architectural decisions and technical considerations behind creating a storage system that can efficiently manage and adapt to continuously evolving datasets, a common challenge in AI development.

6/3/2026

This post details how to run 25 million image vectors on under 1GB of memory in Milvus. It explains why AISAQ and IVF_FLAT indexes failed for this workload and how the simplest FLAT index, combined with FP16 vector storage, mmap for raw vector data, and scalar filtering before vector comparison, achieved low memory usage and good latency. It also provides guidance on when this pattern is applicable and how to interpret Milvus Sizing Tool estimates.

5/7/2026

This post extends the AI Data Storage Engine capability by detailing how Milvus can be integrated with LLM context management tools to combat 'context defocus'. It explains how vector databases and similarity search, exemplified by Milvus, are crucial for semantic code retrieval and cross-session memory in AI agent workflows. The post maps LLM context management challenges (noisy terminal output, tool outputs, codebase navigation, file reads, model verbosity, memory gaps) to potential solutions and highlights the role of vector databases in providing efficient retrieval of relevant information, thereby improving the quality and performance of LLM interactions.

4/30/2026

Introduces Claude Context, an open-source MCP server for code retrieval that uses Milvus. Details its architecture, including AST-based chunking, Merkle tree for incremental updates, and hybrid retrieval. Highlights performance improvements in token usage and tool calls for AI coding agents.

4/30/2026

Introduces RoboBrain, an embodied-intelligence operating system for robot task execution, and details its use of Milvus as the long-term semantic memory tier. Explains how Milvus powers the retrieval of past experiences, exception cases, and operational knowledge through vector similarity search and metadata filtering to improve robot decision-making in dynamic physical environments. Highlights the integration of Milvus with other storage systems for different data types and query patterns.

4/28/2026

This post evaluates and compares three large language models (DeepSeek V4, GPT-5.5, and Qwen3.6-35B-A3B) based on practical tests relevant to AI application development, including live information retrieval, concurrency bug debugging, and long-context analysis. It demonstrates how to connect DeepSeek V4 to Milvus for retrieval-augmented generation (RAG) pipelines, providing a concrete example of integrating external knowledge bases with LLMs. The analysis covers model specifications, performance in specific tasks, and considerations for deployment and cost.

4/24/2026

This post details the integration of Milvus 2.6's hybrid search and tiered storage features with the Hermes AI agent. It explains how hybrid search (combining vector and BM25 full-text search with RRF) addresses the limitations of keyword-only retrieval in Hermes's learning loop, enabling it to connect semantically similar information across sessions. It also highlights how Milvus 2.6's tiered storage allows the retrieval backend to fit on a low-cost VPS, making it practical for personal AI agents. The post provides installation instructions and code examples for setting up Milvus 2.6 standalone and configuring a collection for hybrid search, along with a Python script for performing hybrid searches.

4/21/2026

This post introduces two novel patterns for integrating Milvus with Anthropic's Managed Agents to provide long-term memory. The first pattern enables semantic recall within a single agent session by dual-writing indexed events to a Milvus collection, allowing agents to query their own past decisions and actions using vector search. The second pattern establishes a shared agent memory pool across sessions by partitioning a Milvus collection by tenant, enabling parallel agents to learn from collective experience. The post provides code examples for embedding events, performing session-scoped searches, and utilizing Milvus partition keys for tenant isolation. It also discusses production considerations such as indexing strategy, consistency boundaries, and embedding cost control.

4/17/2026

This post provides a technical deep dive into vector quantization, contrasting the RaBitQ and TurboQuant methods. It explains the core concepts of vector quantization, its application in vector search and KV-cache compression, and the specific technical innovations of RaBitQ (random rotation, optimal quantization estimation). It also critiques TurboQuant's methodology, including benchmarking fairness and attribution of prior work. The post highlights the practical implications for Milvus, including the adoption of IVF_RABITQ and the importance of rigorous evaluation of new techniques for AI infrastructure.

4/17/2026

Introduces 'Vector Graph RAG', a Python library that enables multi-hop reasoning for RAG systems using only Milvus. It details a novel approach to represent graph structures (entities, relations, passages) within Milvus collections using ID references. The post outlines the four-step retrieval flow: seed retrieval, subgraph expansion (one hop by default), LLM reranking of expanded relations, and answer generation. It highlights the performance benefits of avoiding a separate graph database and iterative LLM calls, achieving competitive recall on multi-hop QA benchmarks.

4/9/2026

This post introduces the concept of 'Harness Engineering' as a critical discipline for building reliable autonomous AI agents. It details how Harness Engineering relies on hybrid search (semantic and keyword) for agents to access information from live, evolving knowledge bases. The post outlines specific challenges within Harness Engineering, such as context anxiety and self-evaluation bias, and proposes solutions involving structured documentation, layered architectures, and adversarial agent designs (e.g., Planner, Generator, Evaluator). It highlights the need for retrieval systems that can perform both semantic and exact-match queries concurrently to support agent decision-making.

4/2/2026

This post details the integration of RaBitQ, a vector quantization algorithm, into Milvus 2.6. It explains the technical challenges of making RaBitQ production-ready for a distributed system, including segment-aware codebook management, hardware acceleration for popcount operations using SIMD instructions (AVX-512 VPOPCNTDQ on x86 and SVE on ARM), and eliminating runtime overhead by pre-computing and persisting auxiliary floating-point parameters. This integration allows Milvus to achieve significant memory compression and improved query throughput.

3/26/2026

Introduces the CCKM benchmark (Cross-modal, Cross-lingual, Key information, MRL dimension compression) for evaluating embedding models in production RAG scenarios. Benchmarks 10 embedding models across cross-modal retrieval, cross-lingual retrieval, and key information retrieval, and analyzes dimension compression impact. Identifies Gemini Embedding 2 as a strong all-rounder, Qwen3-VL-2B for cross-modal tasks, and Voyage Multimodal 3.5/Jina Embeddings v4 for dimension compression. Discusses modality gap and semantic understanding as key factors for model performance in RAG.

3/26/2026

Introduces and details Milvus's Change Data Capture (CDC) feature for building a standby cluster, enabling cluster-level high availability. Explains the layered HA strategy (node-level, cluster-level CDC, backup) and provides a hands-on tutorial for setting up primary-standby replication using Milvus CDC and Woodpecker WAL, addressing the critical need for disaster recovery in vector databases.

3/25/2026

This post expands on the AI Data Storage Engine by providing a deep dive into the fundamental principles of data addressing in storage systems. It explains how Milvus, like other systems such as HashMap, HDFS, and Kafka, optimizes data location through computation, caching, and pruning to reduce latency. This provides a foundational understanding of the underlying mechanisms that enable efficient data retrieval in vector databases.

3/25/2026

This post introduces advanced techniques for building smarter RAG systems, focusing on query routing and hybrid retrieval. It proposes a four-node architecture for query classification, rewriting, retrieval strategy selection, and minimal-context generation. It also details how to implement unified hybrid retrieval using Milvus 2.6, which supports dense and sparse vectors in the same collection, and outlines a stage-by-stage evaluation methodology for RAG pipelines.

3/24/2026

Introduces a production pipeline for Generative Engine Optimization (GEO) that integrates OpenClaw, Milvus, and LLMs. Details the two-phase process: Phase 0 for ingesting source material into Milvus, and Phase 1 for expanding topics, semantically deduplicating queries using Milvus, performing dual-collection RAG from Milvus knowledge and article archives, generating articles with LLMs, and writing back to Milvus. Highlights Milvus's role in semantic deduplication, dual-collection RAG, and creating a feedback loop for improved content generation. Demonstrates using Milvus Lite for local development and outlines the skill structure within OpenClaw.

3/23/2026

This post details the implementation of a Corrective Retrieval-Augmented Generation (CRAG) system, enhancing RAG pipelines by adding an evaluation and correction step between retrieval and generation. It highlights Milvus's suitability for CRAG in production due to its multi-tenant isolation (Partition Key), hybrid retrieval capabilities (dense vectors, sparse vectors, scalar metadata filtering with RRF), and flexible schema management (dynamic JSON fields). The post provides a practical example of building a CRAG system using LangGraph middleware and Milvus, addressing common RAG failures like retrieval mismatch, stale content, and memory contamination.

3/20/2026

This post details practical optimization strategies for reducing Milvus infrastructure costs, focusing on memory optimization. It introduces and explains the configuration of MMap (Memory-Mapped Files) and Tiered Storage as methods to significantly reduce memory usage by leveraging local disk and object storage. It also highlights the impact of index selection (e.g., IVF_SQ8 vs. HNSW) and embedding dimensionality on cost. The post provides configuration examples for MMap and Tiered Storage in Milvus.

3/18/2026

Airtable details their experience building and scaling a vector search platform using Milvus for production semantic search. Key contributions include: evaluating and selecting Milvus for its self-hosting, scalability, and multi-tenancy features; designing a partitioning strategy with "one base per partition" and mitigating performance degradation by capping partitions per collection and using multiple collections per cluster; selecting HNSW index for its recall and performance; detailing ingestion and query flows; and addressing operational challenges like deployment via Kubernetes CRD and Milvus operator, observability with infrastructure and service-level metrics, node rotation strategies, cold partition offloading for cost efficiency, and a data recovery plan involving re-embedding commonly used bases.

3/17/2026

This post details the system architecture and implementation steps for building a RAG knowledge base using PaddleOCR, Milvus, and ERNIE. It covers data extraction with PP-StructureV3, text chunking with sliding window overlap, storing embeddings and metadata in Milvus with specific naming conventions and batch insertion, and implementing hybrid search with RRF fusion. Code snippets for parsing, chunking, and inserting documents into Milvus are provided, along with explanations of the benefits of each step.

3/13/2026

This post demonstrates the use of Milvus Lite for building a personal stock monitoring agent. It details the creation of three distinct collections ('decisions', 'preferences', 'patterns') within Milvus to store different types of personal investment data. The post also outlines the implementation of a memory extraction mechanism that uses an LLM to identify and store relevant insights from conversations into the appropriate Milvus collections, including a deduplication step to avoid redundant entries. This showcases Milvus's capability in personal knowledge management and decision support systems.

3/13/2026

This post analyzes the impact of Google's Gemini Embedding 2 on the necessity and application of multi-vector search in vector databases like Milvus. It differentiates between unified multimodal embeddings and multi-vector retrieval, explaining that Gemini simplifies pipelines for data where all modalities describe the same content. However, it argues that Milvus's multi-vector search remains essential for scenarios involving fundamentally different semantic spaces (biometrics, structured attributes) or combining diverse retrieval goals (keyword precision with semantic recall). The post also suggests a hybrid approach where Milvus collections can house both unified Gemini embeddings and additional specialized vector columns.

3/11/2026

This post introduces the integration of Milvus as a vector database within the UltraRAG v2 framework. It provides specific code modifications to the UltraRAG retriever module to support Milvus, including the addition of `retriever_init_Milvus` and `retriever_search_Milvus` tools. The post also outlines the steps for deploying Milvus, setting up the dataset, and configuring UltraRAG to utilize Milvus for retrieval in a RAG pipeline.

3/10/2026

This post details the end-to-end construction of a lightweight RAG chatbot using Milvus and RustFS. It covers the installation of Milvus and RustFS via Docker Compose, including configuration adjustments for object storage. The process of chunking Markdown documentation, generating embeddings with OpenAI's text-embedding-3-large model, and storing these embeddings in a Milvus collection is explained. Finally, it outlines the RAG pipeline, demonstrating how to embed user queries, retrieve relevant document chunks from Milvus using cosine similarity search, construct a prompt, and call an LLM (GPT-5) for an answer. The post provides code snippets for each stage and emphasizes the flexibility of the retrieval layer.

3/6/2026

This post demonstrates a practical implementation of a multimodal RAG pipeline. It details the use of ColQwen2 for encoding PDF pages as images into multi-vector embeddings, Milvus Lite for storing and searching these embeddings, and Qwen3.5 for generating answers based on retrieved page images. The implementation includes environment setup, PDF page conversion to images, encoding and insertion into Milvus with a specific schema (id, doc_id, patch_idx, vector), and a retrieval strategy using MaxSim aggregation for page ranking. The retrieval process involves encoding the query, searching Milvus for each token vector, and aggregating scores by page to identify the most relevant content for the LLM.

3/3/2026

This post details the integration of Milvus into an e-commerce image generation pipeline. It showcases how Milvus's hybrid search capabilities are used to retrieve visually similar bestseller images, which then serve as references for AI image generation models (Nano Banana 2) and style analysis (Qwen 3.5). The post highlights the use of Milvus for finding bestsellers based on a new product's flat-lay photo, leveraging its ability to perform vector search with scalar filtering to match visual styles and compositions. It also discusses the cost and throughput benefits achieved by this retrieval-based approach.

2/28/2026

Introduces 'memsearch' as a solution for persistent, editable AI agent memory to reduce token costs. Discusses the limitations of current AI agent search and memory mechanisms, proposing hybrid search (BM25 + vector search) as a solution. Details tools like index1, QMD, and Milvus for implementing hybrid search at various scales, with a focus on Milvus for team and enterprise deployments.

2/26/2026

This post introduces the integration of Milvus with Google's Agent Development Kit (ADK) to build AI agents with long-term memory. It details how Milvus functions as the backend for ADK's MemoryService, enabling semantic retrieval and hybrid queries for persistent memory. The post provides a practical example of building a technical support agent using Milvus Lite/Standalone and ADK, demonstrating memory storage and retrieval functions, environment setup, and Milvus database initialization with a focus on vector indexing and schema design for AI agent memory.

2/18/2026

This post details the integration of Project_Golem with Milvus 2.6.8 to enable 3D visualization of RAG retrieval for debugging. It highlights the use of Milvus's Streaming Nodes for real-time ingestion and incremental indexing, making RAG systems more observable and scalable. The post also outlines a future path for incremental visualization and the benefits of Milvus's hybrid search capabilities for RAG debugging.

2/14/2026

This post provides a comparative analysis of GLM-5, MiniMax M2.5, and Gemini 3 Deep Think LLMs, evaluating their strengths in areas like system engineering, scientific reasoning, and cost-efficiency. It includes hands-on comparisons for 3D scene rendering and SVG generation. Crucially, it presents a RAG tutorial demonstrating how to wire up GLM-5 with Milvus to build a knowledge base, detailing data preparation, LLM and embedding model setup, and the process of embedding and storing Milvus documentation FAQs in Milvus.

2/10/2026

This post details the integration of Milvus as a memory layer within the Agno multi-agent framework. It explains how Agno's AgentOS runtime layer facilitates production-ready deployment of agents, and how Milvus's native support simplifies memory management, offering flexible deployment modes (Lite, Standalone, Distributed) for scaling. The post also discusses the operational benefits of this layered architecture, including independent scaling of Agno and Milvus, and the implications of shared retrieval results in multi-agent teams.

2/3/2026

This post demonstrates the practical application of Milvus as a semantic memory store for long-running AI agents. It details how Milvus, specifically Milvus Lite for local development and distributed Milvus for production, can be used to embed and retrieve progress records and git commits, providing agents with persistent, cross-session knowledge. The post highlights Milvus's capabilities in handling billions of vectors, supporting hybrid search for combining vector similarity with traditional filtering, and its role in enabling agents to recall relevant past information for complex tasks.

1/30/2026

This post details the local storage design of Claude Code, focusing on its stability and reliability for AI coding assistance. It emphasizes project isolation, immediate data persistence to disk, ordered action history, and easy rollback of code edits. The storage architecture includes a global configuration file (~/.claude.json) and a main data directory (~/.claude/) containing project-specific session data, history, skills, plugins, and file edit history. Configuration is managed through a three-layer model: global, local, and project-level settings, with project-level settings taking the highest priority. This design ensures that Claude Code behaves like a robust local development tool by treating coding sessions as persistent, version-controlled projects rather than ephemeral chat interactions.

1/27/2026

This post analyzes the emergence of long-running agents like Claude Cowork and contrasts their task-driven memory mechanisms with knowledge-driven RAG systems. It details how Claude Cowork likely uses a hybrid approach of persistent long-term memory and on-demand retrieval tools, including semantic and keyword search, to manage state across extended sessions. It also contrasts this with ChatGPT's more structured, layered memory approach (User Memory, Session Metadata, Current Session Messages, and a cross-session summary). The post then outlines the engineering challenges of building writable memory systems for agents, focusing on determining what to remember, how to store and tier memory, and how to implement fast writes without degrading retrieval performance.

1/22/2026

This post demonstrates how to build an AI agent using LangSmith Agent Builder and Milvus. It highlights the use of Milvus as a vector database for providing memory and context to AI agents, enabling them to answer technical questions and remember user preferences. The post details the steps involved in creating an agent, configuring tools (including Milvus's MCP server for vector search), and testing its functionality. It also contrasts Agent Builder with DeepAgents, emphasizing Agent Builder's focus on simplicity and rapid prototyping for AI agent development with integrated memory capabilities.

1/21/2026

Introduces the ScaNN index, detailing its two key innovations: a score-aware quantization objective that preserves the relative ordering of true neighbors and FastScan, a SIMD-optimized 4-bit PQ lookup path for reduced memory bottlenecks. Provides experimental results demonstrating ScaNN's performance advantages over IVFFLAT and IVF_PQ in terms of QPS and memory footprint, particularly when raw data is not loaded.

1/19/2026

Introduced and open-sourced a Semantic Highlighting model (zilliz/semantic-highlight-bilingual-v1) for RAG context pruning. The model is an encoder-only network based on BGE-M3 Reranker v2, trained to score tokens and aggregate scores to sentence level for relevance filtering. It was trained on over 5 million bilingual samples generated using LLM annotation with reasoning, and evaluated to achieve state-of-the-art performance. The model aims to reduce token usage and improve LLM inference by filtering out irrelevant sentences from retrieved documents.

1/14/2026

This post evaluates RustFS as an S3-compatible object storage backend for Milvus, exploring its architecture and compatibility. It provides a hands-on guide to replacing MinIO with RustFS in a Milvus deployment and discusses experimental storage performance testing.

1/9/2026

This post explains how Milvus 2.6's features (hybrid dense-sparse retrieval, scalar filtering, tiered storage) enable on-demand conversational retrieval, a key component of Claude's memory system. It details the technical demands of such systems, including low latency, hybrid search, and storage-compute separation, and positions Milvus as an enabling technology for these capabilities.

1/6/2026

Introduces and open-sources a new bilingual semantic highlighting model (zilliz/semantic-highlight-bilingual-v1) specifically trained for production RAG and AI search. This model addresses limitations of existing semantic highlighting solutions, such as small context windows, poor out-of-domain generalization, weaker multilingual performance, and restrictive licensing. The post details the shortcomings of keyword-based highlighting and existing semantic highlighting approaches (OpenSearch semantic highlighter, Provence/X Provence, Open Provence) and explains the design considerations for the new model, emphasizing strong multilingual performance, large context window, robust generalization, high precision, and a permissive license.

12/29/2025

Introduces and explains the new Phrase Match feature in Milvus 2.6, which enables full-text search with positional indexing and a 'slop' parameter for flexible phrase matching. Details the technical implementation using Tantivy, the schema configuration required for VARCHAR fields (`enable_analyzer`, `enable_match`), and provides examples of how to use `PHRASE_MATCH` with varying `slop` values to filter search results before vector similarity ranking.

12/24/2025

Introduces Max-Min Semantic Chunking, a new strategy for RAG pipelines that embeds sentences first and then uses semantic similarity to define chunk boundaries. This method contrasts with traditional fixed-size or recursive chunking by prioritizing semantic coherence over arbitrary length. The post details the six-step process of Max-Min Semantic Chunking, its strengths (dynamic boundaries, practical tuning, low overhead), and its limitations (potential for context fragmentation in long documents). It also references Milvus as a suitable vector database for storing embeddings and discusses the impact of chunking quality on RAG system performance.

12/23/2025

This post elaborates on context engineering strategies to combat context rot in AI agents. It details Just-in-Time (JIT) retrieval and pre-retrieval using vector search, highlighting Milvus as a key component for the latter. The post introduces a hybrid approach combining both JIT and vector retrieval, explaining how Milvus's features like vector search with scalar filtering and incremental updates support this architecture. It provides guidance on choosing the right approach based on knowledge stability and task predictability, and discusses scenarios where context windows might still be insufficient.

12/19/2025

This post details how Milvus can be integrated with LangChain 1.0 to provide persistent, intelligent memory for agents. It explains how Milvus's vector database capabilities enable efficient storage, retrieval, and semantic similarity search of information, going beyond LangChain's core reasoning and tool-calling functionalities. The post highlights Milvus as a scalable and high-performance memory layer for production-ready agents.

12/16/2025

Introduces the Ngram Index for Milvus, enabling faster LIKE queries and keyword matching by indexing n-grams of text. Details the index construction (min_gram, max_gram) and query acceleration phases, including handling of infix and wildcard matches. Discusses limitations like increased index size and evaluates performance benchmarks.

12/15/2025

Introduces and details Milvus 2.6's Tiered Storage architecture, which enables on-demand loading of hot and cold data. Explains the inefficiencies of the previous full-load model and how Tiered Storage addresses them through lazy loading, partial loading (tenant-aware vector index loading, column-level scalar field loading), and LRU cache eviction. Provides performance and cost-saving evaluations.

12/10/2025

Introduces AISAQ, a new disk-based vector index for Milvus that achieves a 'Zero-DRAM-Footprint Architecture' by storing PQ codes and other search-critical data on disk. This significantly reduces memory usage for billion-scale vector search workloads, offering a 3,200x reduction in memory footprint compared to previous approaches while maintaining performance. The post explains the underlying principles of graph-based vector search, the trade-offs involved, and how AISAQ builds upon DISKANN by optimizing disk data layout for efficient I/O.

12/10/2025

Introduces and details the hybrid GPU-CPU approach for CAGRA indexing in Milvus 2.6.1. Explains the CAGRA algorithm (NN-Descent and 2-hop pruning) and its GPU acceleration. Details the `adapt_for_cpu` parameter and its four execution modes for balancing GPU construction with CPU querying. Presents experimental results demonstrating significant index build time improvements (12-15x) and discusses query performance and recall accuracy.

12/8/2025

Introduces the Geometry Field (DataType.GEOMETRY) for storing spatial data (Point, LineString, Polygon) using WKT format. Implements an RTREE index for efficient spatial filtering through coarse and fine filtering stages, supporting operators like st_contains, st_within, and st_dwithin. Enables joint retrieval by combining RTREE-based spatial filtering with vector similarity search, allowing for spatially aware intelligent retrieval.

12/5/2025

Introduces and explains Milvus's new Array of Structs field type, which allows a single record to store an ordered list of Struct elements, each capable of holding multiple vectors and scalar fields. Details the new MAX_SIM scoring metric, which computes entity-level similarity by taking the maximum similarity score between query vectors and all vectors within an Array of Structs field for a given entity. Demonstrates how these features together enable true entity-level retrieval, solving issues like fragmented search results and duplicate entities in RAG, e-commerce, and other multi-vector applications.

12/4/2025

Introduces JSON Shredding, a new feature that optimizes the storage and querying of semi-structured JSON data in Milvus. It categorizes JSON keys into typed, dynamic, and shared, storing them in optimized columnar formats (Arrow/Parquet) or a compact binary-JSON column with an inverted index for shared keys. This enables schema-free agility with high query performance, demonstrated by benchmarks showing up to 88.9x speedup for shared key queries and 15-30x for typed key queries.

12/3/2025

Introduces the Embedding Function (Data-in, Data-out) in Milvus 2.6, a new built-in capability that handles vectorization internally by connecting to external model providers. This feature allows users to insert raw data and query using raw text, with Milvus automatically managing the embedding generation process. It details the 'Function' module framework, the workflow steps, configuration via milvus.yaml, and schema definition including scalar and vector fields, and the embedding function itself. Demonstrates insertion of raw text and performing semantic search using text queries.

12/2/2025

This post chronicles the 7-year journey of Milvus, highlighting two major rebuilds and its growth to over 40,000 GitHub stars. It details key milestones including the open-sourcing of Milvus 0.10, joining the LF AI & Data Foundation, winning the BigANN challenge, and the ground-up reinvention for Milvus 2.0. It then focuses on the significant advancements in 2025 with Milvus 2.5 introducing native hybrid search and Milvus 2.6 delivering substantial performance and cost improvements through features like tiered storage, 1-bit quantization (RaBitQ), an improved full-text engine, Path Index for JSON, AiSAQ, Geospatial Search with R-Tree, CAGRA+ Vamana, a streamlined data workflow, and increased collection support. The post also introduces two new open-source tools: DeepSearcher for local deep research and Claude Context for enhanced AI coding assistants. Finally, it emphasizes Milvus's adoption by over 10,000 teams in production for various AI workloads, citing performance metrics like sub-50ms retrieval and multi-tenant capabilities.

11/28/2025

This post details Reddit's process for selecting a vector database for ANN search, evaluating Milvus 2.4 alongside other solutions. It highlights Milvus's strengths in hybrid search, metadata filtering, support for multiple indexing methods, and its open-source nature as key factors in their decision. The evaluation process involved qualitative and quantitative assessments against functional and non-functional requirements, emphasizing scale, reliability, community, and expressive metadata types.

11/25/2025

This post details the integration of Milvus with the RAG-Anything framework to build a multimodal RAG system. It explains how RAG-Anything's '1 + 3 + N' architecture parses and processes diverse content types (text, images, tables, equations) in parallel, and how Milvus serves as the scalable vector search backend for retrieving evidence across these modalities. It also provides a hands-on demo with code snippets and configuration for building a multimodal Q&A system using Milvus Lite, TongYi embedding model, and Alibaba Cloud services.

11/24/2025

This post details the integration of Milvus with the OpenAgents framework to provide a scalable, high-performance long-term memory layer for multi-agent systems. It highlights how Milvus enables semantic search, billion-scale horizontal scalability, and multi-tenant isolation through Partition Key for agent memory. The post also provides a practical example of building a developer-support community chatbot using OpenAgents and Milvus, demonstrating how agent conversations, past solutions, and user queries are stored and retrieved via vector embeddings.

11/20/2025

This post details the integration of Gemini 3 Pro with Milvus to build a more robust RAG system. It highlights Gemini 3 Pro's advanced reasoning and multimodal capabilities, and explains how Milvus serves as a fast, scalable retrieval layer. The post provides a practical guide to setting up a RAG pipeline, including prerequisites, dataset preparation (using Milvus documentation FAQs), LLM and embedding model setup (gemini-3-pro-preview and text-embedding-004), and loading data into Milvus using Milvus Lite. It demonstrates creating a collection with a dynamic schema and inserting text data with generated embeddings, showcasing the flexibility of Milvus's dynamic fields.

11/17/2025

Introduces the deployment of NVIDIA RDMA (Remote Direct Memory Access) to Cloudian's S3-compatible storage. This integration aims to dramatically improve storage performance for AI applications by enabling direct memory access between compute nodes and storage, bypassing the CPU for data transfers. The post highlights the benefits for AI storage performance, suggesting a significant uplift in data throughput and reduced latency.

11/5/2025

This post introduces Parlant, an agent framework that uses Alignment Modeling with Granular Guidelines. It details how Milvus is integrated to perform semantic retrieval of these guidelines, enabling dynamic context injection for LLM agents. The post provides code examples for defining guidelines and interacting with Milvus for guideline matching and user memory storage, highlighting Milvus's role in achieving low-latency, high-accuracy retrieval for controlled and explainable agent behavior.

10/27/2025

This post provides a deep dive into the Inverted File (IVF) vector index, a core component for Approximate Nearest Neighbor (ANN) search in vector databases. It explains the IVF index's architecture, including clustering, centroids, and inverted lists. It details the index building process involving K-means clustering, vector assignment, and optional compression techniques like Scalar Quantization (SQ8) and Product Quantization (PQ). The post also outlines the search process, emphasizing the role of nprobe in balancing recall and latency. Practical advice is given on choosing nlist and tuning nprobe for different scales and performance requirements, along with a comparison of IVF variants (IVF_FLAT, IVF_PQ, IVF_SQ8).

10/17/2025

This post introduces the integration of Milvus as a semantic cache layer with the vLLM Semantic Router. It details how Milvus's distributed vector database capabilities enable efficient semantic caching for AI systems by storing and retrieving embeddings, thereby improving scalability, reducing latency, and lowering costs. The post explains the architecture of semantic routing, mixture of models, and the cache layer, emphasizing Milvus's role in handling large-scale semantic search with ANN algorithms. It also provides a practical guide for developers to test this integration locally, demonstrating the caching mechanism for AI applications.

10/16/2025

Introduces and details the Milvus Analyzer, a new component for text preprocessing and tokenization that enables efficient full-text search. Explains the Tokenizer and Filter components, various Analyzer types (Standard, English, Chinese, Multi-language, Custom), and provides Python SDK examples for their usage. This enhances Milvus's hybrid search capabilities by improving keyword matching.

10/11/2025

Introduces and implements a 'Late Chunking' strategy for RAG systems. This involves embedding entire documents first to capture global context, then averaging contiguous token embeddings to form chunk vectors. Provides Python code for `sentence_chunker`, `document_to_token_embeddings`, and `late_chunking` functions. Demonstrates comparison with traditional chunking methods using cosine similarity and shows how to store and query these embeddings in Milvus.

10/3/2025

This post extends the understanding of the AI Data Storage Engine by detailing the critical role of embedding models in RAG systems, which are often used in conjunction with vector databases like Milvus. It provides a comprehensive guide on evaluating and selecting embedding models based on factors such as context window, tokenization, dimensionality, vocabulary size, training data, cost, MTEB scores, and domain specificity. This information is crucial for users to effectively leverage Milvus for RAG applications by ensuring relevant data is retrieved based on high-quality embeddings.

9/30/2025

This post introduces the concept of 'Context Offloading' as a strategy for managing long contexts in AI systems. It specifically highlights the role of vector databases like Milvus in persisting non-critical data and fetching it on demand, thereby reducing the cognitive load on LLMs. This contributes to the broader theme of how Milvus supports AI pipelines by providing efficient data storage and retrieval for context management.

9/11/2025

This post introduces the langgraph-up-react template and demonstrates its use with Milvus for building agentic RAG systems. It explains how Milvus serves as the vector database for indexing and retrieval, enabling enterprise knowledge base Q&A. The post also touches upon Milvus deployment options and performance tuning considerations within the context of AI agent development.

9/4/2025

This post demonstrates building a text-to-image retrieval engine using CLIP for embeddings and Milvus for vector storage and search. It provides code examples for installing dependencies, initializing Milvus client, loading CLIP model, defining feature extraction functions, creating a Milvus collection, processing and inserting image data, and performing text-based image searches with visualization. It also introduces the concept of integrating this retrieval system with Nano-banana for multimodal RAG applications.

8/25/2025

Introduces Claude Context, an open-source MCP plugin that adds semantic code search capabilities to AI coding assistants. It details the system architecture, including the use of Zilliz Cloud (Milvus) as the vector database and support for various embedding models. Key technical challenges addressed include intelligent code chunking using AST-based parsing and LangChain's text splitter, and efficient handling of code changes through a Merkle Tree-based synchronization mechanism. The post highlights the benefits of this approach over grep-only methods, such as reduced token usage and improved search accuracy.

8/22/2025

This post contributes a decision framework for selecting vector databases, detailing criteria across functionality (vector data types, indexing, retrieval, architecture), performance (metrics, benchmarking), and ecosystem (AI integration, tooling, commercial balance, deployments). It specifically mentions Milvus as a leading option and references VDBBench for practical benchmarking.

8/11/2025

This post demonstrates the integration of Milvus with Refly and MCP-Milvus-Server to build AI-powered content creation workflows. It details the setup process for Milvus, Refly, and the MCP server, and provides examples of using natural language to query Milvus via MCP-Milvus-Server. It also illustrates building a content deployment guide using Refly's workflow orchestration, highlighting multi-modal capabilities.

8/8/2025

This post analyzes GPT-5's impact on RAG and agent development, emphasizing its improved accuracy, multimodal I/O, and cost reductions. It details how the 400k context window and better tool use enhance RAG pipelines and agent workflows. The post also argues for the continued relevance of vector databases like Milvus, stating that long context windows are not a replacement for structured retrieval and secure data management.

8/7/2025

This post demonstrates a practical application of Milvus within a RAG system, specifically integrating the newly open-sourced GPT-oss models. It provides a hands-on tutorial with code examples for setting up Milvus, embedding documents, and performing similarity searches to retrieve context for the LLM. The post highlights the use of Milvus's embedding capabilities and its role in providing relevant data for AI reasoning.

7/25/2025

This post introduces a practical application of Milvus within a chatbot architecture, integrating Kimi K2 for reasoning and OpenAI for embeddings. It provides concrete Python code for a `VectorDatabase` class handling Milvus connection, collection creation (with IVF_FLAT index and COSINE metric), document addition, and similarity search. It also outlines a `SmartAssistant` class that orchestrates these Milvus operations using specialized tool functions, showcasing a full RAG pipeline for chatbots.

7/24/2025

Introduces Code Context, an open-source semantic code search engine that leverages Milvus for vector storage. Details its architecture, including AST-based code chunking, live incremental indexing, and integration with AI coding agents via the MCP protocol. Provides examples for integrating with existing tools (Claude Code, VSCode) and building custom AI assistants using Milvus and embedding models. Highlights private deployment capabilities and real-time indexing mechanisms.

6/30/2025

This post introduces the integration of Qwen3 embedding and reranking models with Milvus to build a RAG system. It details the setup of Qwen3 embedding and reranking models, including their multilingual capabilities, instruction prompting, variable dimensions, and context length. The post provides a practical implementation of a two-stage retrieval pipeline: dense retrieval using Qwen3 embeddings (with query-specific prompting) and reranking using Qwen3 cross-encoder. It also demonstrates data preparation, loading into Milvus Lite, and performing searches and reranking. The use of Milvus's Inner Product (IP) metric and strong consistency level is highlighted.

6/27/2025

This post chronicles the historical development of Milvus, from its initial conception as a solution for storing and searching vector embeddings to the significant architectural shift with Milvus 2.0. It details the motivations behind rebuilding the system with a decoupled storage-compute architecture to address enterprise needs for scalability and operational simplicity. The post also discusses the impact of RAG on Milvus's growth and the strategic decision to launch Zilliz Cloud as a managed service, highlighting its performance improvements and security features.

6/23/2025

This post provides a comprehensive tutorial on building a production-ready AI assistant by integrating Spring Boot, Milvus, and Ollama. It details the document ingestion pipeline using TikaDocReader and text splitting, generating embeddings, and storing them in Milvus. The RAG workflow is explained with an example of retrieval and memory integration using Redis. It also covers API security with Spring Security's RBAC and observability using OpenTelemetry and Micrometer, including specific configurations for tracing gRPC spans and exposing Prometheus metrics. The post includes runnable code examples and deployment instructions.

6/12/2025

Milvus 2.6 introduces RaBitQ 1-bit quantization for significant memory reduction (up to 72%) and performance gains (4x QPS) without compromising recall. It also enhances full-text search with a revamped multi-language analyzer, adds Phrase Match for nuanced word order search, and introduces Time-Aware Decay Functions for relevance ranking based on age. A new Function interface integrates embedding models directly into the data pipeline for a streamlined 'Data-In, Data-Out' experience. The architectural evolution includes replacing Kafka/Pulsar with a new Woodpecker WAL system for improved performance and reduced operational complexity, and introduces tiered storage for cost reduction.

5/30/2025

This post introduces the use of ONNX for exporting neural network models (including embedding models) and their tokenizers from Python to formats usable in other languages like Java. It provides a workflow for converting HuggingFace models to ONNX, including handling pooling and normalization layers, and demonstrates how to perform inference using ONNX Runtime in Java. This enables the integration of Milvus with AI models deployed in non-Python environments, addressing production constraints.

5/30/2025

This post argues for the obsolescence of SQL in AI applications and positions vector databases like Milvus as the future. It contrasts SQL's limitations with unstructured data and semantic search against the capabilities of vector databases, highlighting Milvus's native multimodal support, agent-friendly architecture, and semantic intelligence. It also presents benchmark data showing Milvus's superior performance over PostgreSQL with pgvector for vector similarity search.

5/21/2025

This post provides a detailed explanation of the Hierarchical Navigable Small Worlds (HNSW) algorithm, a key indexing strategy for vector databases. It covers the underlying Navigable Small Worlds (NSW) concept, the construction and search processes for both NSW and HNSW, and the hierarchical modifications that improve search efficiency and convergence. The post emphasizes HNSW's role in enabling efficient similarity search for large-scale vector datasets and links to a pedagogical Python implementation.

5/20/2025

Introduces and explains the DiskANN index type for Milvus. Details its architecture, how it leverages SSDs for large-scale vector similarity search, and its trade-offs compared to in-memory methods. Explains the Vamana algorithm for graph construction and the use of product quantization for efficient search guidance. Provides Python code examples for creating collections with the DiskANN index and mentions tunable parameters.

5/17/2025

Milvus 2.6 introduces RaBitQ 1-bit quantization with adjustable Refine optimization (SQ4/SQ6/SQ8) for significant memory reduction and performance improvements. It also enhances full-text search with a revamped multi-language analyzer (Lindera, ICU, enhanced Jieba), adds Phrase Match with a 'slop' parameter for word order control, and introduces Time-Aware Decay Functions (exponential, Gaussian, linear) for relevance ranking based on document age. A new Function interface allows direct integration of embedding models (OpenAI, AWS Bedrock, Google Vertex AI, Hugging Face) for streamlined data ingestion and vectorization. Architecturally, a hot-cold tiered storage system is introduced for cost-effective scaling, and a new Streaming Node enables real-time vector processing with direct integration to streaming platforms.

5/16/2025

Introduces and details the implementation of MinHash LSH indexing in Milvus 2.6 for approximate data deduplication. Explains the underlying concepts of MinHash (shingling, hashing) and LSH (banding technique) for efficient near-duplicate detection. Describes the native integration within Milvus, including the MINHASH_LSH index type, JACCARD metric, and distributed architecture support. Provides practical guidance on generating MinHash signatures and using Milvus SDK for schema creation, index building, and data insertion for deduplication tasks.

5/13/2025

Introduces and details the integration of RaBitQ, a novel 1-bit vector compression technique, into Milvus. Explains the engineering challenges and tradeoffs in implementing RaBitQ within Milvus's Knowhere search engine, including pre-computation of auxiliary data and hardware acceleration using CPU instructions (VPOPCNTDQ for AVX512). Describes query optimization techniques like scalar quantization on query vectors and refinement. Introduces the new `IVF_RABITQ` index type in Milvus 2.6, which combines RaBitQ with IVF clustering. Provides usage examples and benchmarking results showing a 3x QPS increase with comparable accuracy.

5/12/2025

This post details how Milvus and Zilliz Cloud implement optimizations for efficient metadata filtering during vector searches. It covers graph index optimization using the Alpha strategy and dynamic fallback to brute-force scans for highly selective filters. It also introduces metadata-aware indexing with specialized subgraphs (column graphs) and iterative filtering inspired by VBase, which processes data in batches to reduce filter operation costs. Additionally, it describes external filtering, a client-side SDK solution for hybrid data systems that reverses the traditional workflow by performing vector search first and then applying custom filters to batches of candidates.

4/30/2025

This post demonstrates building a practical RAG system using Milvus and the Qwen 3 language model. It provides hands-on code for setting up the environment, preparing data by downloading and parsing Milvus documentation, initializing a Milvus collection with appropriate dimensions and consistency level, and inserting document embeddings. It also details the process of performing semantic search queries against the Milvus collection and retrieving relevant text chunks to augment the language model's responses. The post highlights Milvus's role in efficient semantic search for RAG applications.

4/28/2025

This post demonstrates the integration of Milvus with Dify to build a RAG-powered document assistant. It details the steps for setting up Dify and Milvus containers, configuring Milvus as the vector store in Dify, uploading documents and processing them into embeddings, and creating a RAG application by connecting the knowledge base and LLM. The post emphasizes Milvus's role in providing efficient embedding storage and retrieval for RAG systems.

4/15/2025

Introduces and details the implementation of Sparse-BM25 full-text search in Milvus 2.5. Explains the BM25 algorithm and its transformation into an Inner Product calculation for sparse vectors. Details Milvus's novel approach using dynamic statistics (avgdl, term distribution) and search-time encoding to handle the complexities of BM25 computation internally, removing client-side burden. Provides a PyMilvus example for defining a collection with a text field and a BM25 function, creating an AUTO_INDEX with BM25 metric, inserting raw text, and performing searches.

4/9/2025

This post provides a comprehensive guide on selecting the right embedding model for AI applications. It outlines a six-step framework covering: defining task and business requirements, evaluating data characteristics (modality, domain specificity, embedding type), researching available models (popularity, licensing), evaluating candidate models (accuracy, benchmark performance, load testing), planning model integration (weights, hosting, API design), and conducting end-to-end testing. It emphasizes matching model capabilities to specific use cases and data, and highlights the importance of rigorous evaluation beyond benchmark scores.

4/3/2025

This post demonstrates the integration of Mistral OCR with Milvus for advanced document parsing and semantic understanding. It showcases how Mistral OCR can extract text and structure from complex documents, including handwritten notes and invoices, and how this extracted information can be inserted into Milvus for semantic search. The post provides code examples for using Mistral OCR and MilvusClient, and discusses the benefits of this combination for building RAG systems for document analysis.

4/1/2025

This post demonstrates the integration of Milvus with GPT-4o for generating curated images. It details the technical steps involved: setting up dependencies, preparing a dataset of images, defining a feature extractor using ResNet-50 from the `timm` library to generate embeddings, creating a Milvus collection with `COSINE` metric type and a vector dimension of 2048, inserting image embeddings along with filenames into Milvus, and performing similarity searches using a query image. The post also provides example prompts and outputs for generating Ghibli-style images and comic strips based on the search results. It highlights the use of Milvus Lite for local setup and suggests scaling to a Milvus server for larger datasets.

3/25/2025

This post demonstrates integrating the DeepSeek v3-0324 LLM with Milvus for building a Retrieval-Augmented Generation (RAG) system. It details the steps for setting up the environment, preparing data from Milvus documentation, configuring Milvus Lite for local storage, creating a collection with appropriate dimensions and metric types, embedding documents using Milvus's default embedding function, and performing searches. The core contribution is the practical application of DeepSeek v3-0324's advanced reasoning and context window capabilities within a Milvus-powered RAG pipeline, highlighting the ease of integration via OpenAI-compatible APIs and the effectiveness of Milvus for data retrieval.

3/24/2025

This post provides a foundational overview of vector databases, explaining their core concepts, mathematical underpinnings (vector spaces, embeddings, distance metrics), and architectural components (storage, index, query, service layers). It details the vector search workflow from data transformation to retrieval and touches upon consistency challenges in distributed systems.

3/23/2025

This post introduces DeepSearcher, an open-source project implementing an agentic RAG framework. It details the architecture with a Data Ingestion Module (parsing, embedding, vector storage via Milvus) and an Online Reasoning and Query Module that uses iterative retrieval and reflection for complex queries. The post contrasts agentic RAG with traditional RAG, highlighting benefits like proactive problem-solving and dynamic multi-turn retrieval. It provides examples of its application in overview-type queries and discusses its transparency through logging of search and reasoning steps.

3/18/2025

This post elaborates on Milvus's automated sharding and scaling mechanisms, contrasting them with the drawbacks of manual sharding in vector databases. It details Milvus's segment-based architecture, two-layer routing, and query processing flow to explain how it achieves seamless scalability without operational burden. It also mentions the upcoming Milvus 3.0 feature of dynamic shard splitting.

3/17/2025

This post provides a detailed walkthrough of the internal data flow for insert and search operations in Milvus. It elaborates on the lifecycle of a datum from insertion through proxy nodes and message queues to data nodes, segments, and chunks in persistent storage. It also describes the processes of segment sealing, merging, compaction, and index building by index nodes, and how query nodes retrieve and process data for search. The key takeaway is the separation of concerns across node types and the separation of storage and compute, enabling independent scaling.

2/25/2025

This post demonstrates a practical application of Milvus's AI data storage capabilities by integrating it with Browser-Use and Pixtral to filter web content. It showcases how Milvus can store and query both vector embeddings and sparse vectors (for full-text search) from diverse web content. The post specifically highlights the utility of dynamic fields in Milvus for handling unstructured or schema-evolving data, providing a code example for schema creation and data insertion with dynamic fields enabled.

2/21/2025

Introduces DeepSearcher, an open-source project that enhances agentic RAG capabilities with query routing, conditional execution flow, and web crawling. It details the architecture and implementation of these features, including specific Python code for query routing and reflection prompts. The post also discusses the performance benefits of using specialized hardware for LLM inference.

2/6/2025

This post details the construction of a research agent prototype using Milvus as the vector database. It showcases how Milvus is used to store chunks of Wikipedia articles, enabling Retrieval-Augmented Generation (RAG) queries. The implementation involves using Milvus's LangChain integration for efficient data storage and retrieval, demonstrating a practical application of the vector database in an AI agent workflow.

2/3/2025

This post details the implementation of a multimodal semantic search application using Milvus. It showcases the use of the Visualized BGE embedding model to jointly embed text and images into a shared vector space, enabling semantic search across these modalities. The post outlines the indexing process, where product images are embedded and stored in Milvus. It then describes the retrieval step, where a multimodal query (text + image) is used to perform a similarity search. Finally, it details a reranking step using Phi-3 Vision to improve the relevance of the retrieved results, demonstrating a complete pipeline for multimodal search.

11/25/2024

This post introduces the Hierarchical Navigable Small Worlds (HNSW) algorithm and its implementation in the HNSWlib library. It explains the core concepts of HNSW, including its hierarchical layers and navigable small worlds structure, and details how HNSWlib provides efficient Approximate Nearest Neighbor (ANN) search capabilities. The post includes a step-by-step guide on setting up HNSWlib, preparing data, building an index with key parameters (`space`, `dim`, `max_elements`, `ef_construction`, `M`, `ef`), and performing nearest neighbor searches using `knn_query`. Finally, it compares HNSWlib with purpose-built vector databases like Milvus, noting HNSWlib's suitability for prototyping and medium-scale datasets versus Milvus's advantages for larger, enterprise-level requirements.

10/30/2024

This post introduces Matryoshka embeddings, a technique that embeds multiple scales of representation within a single vector, allowing for flexible granularity to balance precision and computational cost. It details the 'funnel search' approach for accelerating similarity searches and explains the training objective modification for Matryoshka embeddings. Milvus seamlessly supports Matryoshka embedding models, with examples like OpenAI's text-embedding-3-large, Nomic's nomic-embed-text-v1, and Alibaba's gte-multilingual-base.

5/30/2024

Introduces Milvus Lite, a new product offering that allows Milvus to run locally within a Python application. This is achieved by reusing core Milvus components for vector indexing and query parsing while removing elements for distributed systems. It integrates with AI development stacks like LangChain and LlamaIndex, and uses an SQLite file for persistence. The API is compatible with the standard Milvus API, with the primary difference being the `uri` parameter pointing to a local file.

3/8/2024

Introduced support for Apache Parquet file imports in Milvus version 2.3.4. This feature enhances data processing efficiency by leveraging Parquet's columnar storage, compression, and encoding for optimized storage and query performance. The post details how to import Parquet files directly from object storage (e.g., MinIO) and how to use the BulkWriter tool, which now defaults to Parquet output, to generate Parquet files for import. Future plans include optimizing large file imports by splitting scalar and vector columns.

1/5/2024

This post highlights Milvus's 2023 achievements, focusing on engineering advancements. Key technical contributions include: zero downtime during rolling upgrades (refined from v2.2.3), a 3x performance improvement in production environments (especially with filtered searches and streaming insert/search), a 5% recall improvement on the Beir dataset using hybrid search (dense + sparse embeddings) and a reranking model, and 10x memory saving on large datasets through disk-based indexing using memory-mapped files (MMap) introduced in Milvus 2.3.4. Other technical developments mentioned are dynamic schema (v2.2.9), Upsert, Range Search, and Cosine metrics (v2.3), Partition Key for multi-tenancy, and enhanced memory management, coroutine handling, and CPU optimization in v2.3.4 for B2B scenarios. The post also mentions the development of VectorDBBench for benchmarking and Pyanns for sparse embedding search.

12/15/2023

This post demonstrates the integration of Milvus with NVIDIA Merlin for efficient vector similarity search in recommender workflows. It highlights Milvus's GPU acceleration capabilities, leveraging NVIDIA RAFT, to achieve significant speedups (37x to 91x) in ANN search for user and item embeddings generated by Merlin Models. The technical details cover the use of NVTabular for data preprocessing, Merlin Models for training Two-Tower deep learning models, Milvus for building GPU-accelerated indexes and performing similarity searches, and NVIDIA Triton Inference Server for the inference stage. The post also touches upon Milvus's system design, separating compute and storage for scalability, and its reliance on underlying libraries like cuDF and RAFT for GPU acceleration.

12/8/2023

This post provides a comprehensive introduction to vector embeddings, explaining their creation, meaning, and generation for various data types (images, text, audio, video, multimodal). It details how to extract embeddings from deep learning models by removing the last layer and discusses popular open-source models and libraries for generating these embeddings. While it mentions using vector databases like Milvus and Zilliz Cloud, it does not introduce new engineering capabilities or significant updates to existing ones within Milvus itself. The technical depth is focused on the embedding generation process rather than Milvus's internal systems.

11/20/2023

Introduced support for Array data types for enhanced filtering in search. Enabled complex delete expressions for advanced data management. Integrated TiKV as an alternative to Etcd for scalable metadata storage. Added interface-level support for FP16 vector types. Improved rolling upgrade experience with reduced query impact. Implemented performance optimizations for data loading, inserts, and recall phase. Permanently deleted TimeTravel-related code and deprecated MySQL as a metadata store.

11/15/2023

Introduced MMap (memory-mapped files) as a beta feature to redefine data management and increase storage capability. MMap maps large files directly into the system's memory space, transforming files into contiguous memory blocks and eliminating explicit read/write operations. This allows Milvus to process more data within limited memory, especially benefiting users less sensitive to performance fluctuations. Configuration involves adding `mmapDirPath` to `queryNode` in `milvus.yaml`. The feature optimizes data access based on locality, writing scalar data to disk for sequential segments and indexing variable-length data with an offsets array. Vector data uses MMap selectively while retaining adjacency lists in memory. This results in a copy-free streaming process, drastically reducing memory overhead and enabling Milvus to efficiently handle double the data volume.

11/9/2023

This post compares specialized vector databases (like Milvus) with vector search libraries (like FAISS) and vector search plugins for traditional databases. It details the technical differences in architecture, scalability, and feature sets, emphasizing why Milvus is a more optimal solution for unstructured data storage and retrieval compared to libraries and plugins, particularly for large-scale and real-time applications.

10/16/2023

This post details the MLOps challenges and strategies for reliably ingesting unstructured data at scale for RAG applications. It highlights the complexities of diverse file formats, lack of schema, and varying file sizes. The post introduces the concept of a vector embedding pipeline (e.g., VectorFlow) to streamline the process of connecting raw data to vector databases like Milvus, covering chunking, metadata orchestration, embedding, and upload. It also outlines a 5-step experimental approach for evaluating RAG pipeline parameters, emphasizing the use of a small, familiar dataset, standardized questions, an automated evaluation system, distinct LLMs for evaluation and embedding, and averaging scores to mitigate stochasticity. The post concludes by emphasizing how vector embedding pipelines and Milvus enable efficient testing across environments and a smooth transition to production.

8/28/2023

Milvus 2.3 introduces GPU indexing using the RAFT algorithm via Knowhere, leading to 3-10x faster QPS. It adds Arm64 support for Docker images. The upsert operation is implemented as a delete-then-insert combination. Range search allows specifying distance bounds for queries. ScaNN index is now supported for faster queries. The Growing Index automatically indexes streaming data. An iterator interface in Pymilvus enables batch retrieval of over 16,384 entities. MMap support increases single-machine data capacity. CDC is introduced for data synchronization and backup. Performance improvements include optimized HNSW filtering, full multi-core CPU utilization, a refactored stateless QueryNode, and a new message queue system using NATS and Bookkeeper. The load balancer is optimized for faster fault detection and adjustment.

6/6/2023

This post details the integration of Milvus as a vector store for LangChain's conversational memory. It outlines the setup process involving LangChain libraries, OpenAI embeddings, and Milvus Lite. The core contribution is demonstrating how to use `VectorStoreRetrieverMemory` with Milvus to store and retrieve conversational context, enabling LLMs to maintain state and provide more coherent interactions. It provides code examples for initializing Milvus Lite, creating a retriever, saving conversation context, and querying the memory within a `ConversationChain`.

6/6/2023

Introduced support for JSON data types, allowing storage and querying of JSON alongside vector metadata. Enabled dynamic schema functionality via SDK, automatically filling dynamic fields into a hidden JSON field. Implemented Partition Key feature, allowing user-specific columns as primary keys for partitioning, removing partition limits and the need for explicit partition load/release APIs. Added support for Alibaba Cloud OSS.

8/5/2022

Milvus 2.1 introduces significant performance enhancements including a new routing protocol reducing retrieval latency to 5ms, a new cost evaluation model and concurrency scheduler for a 3.2x performance boost with small-nq queries, and in-memory replicas for improved scalability and availability. Data loading is accelerated by Zstandard compression and goroutine pools. String data types (VARCHAR) are now supported as scalar data, usable as primary keys and for attribute filtering, with MARISA-Tries used for scalar inverted indexes to improve filtering speed. Kafka is added as an alternative message queue to Pulsar. A production-ready Java SDK is released, and observability is improved with new monitoring metrics and optimized log keeping. Embedded Milvus is introduced for simplified local development.

7/8/2022

This post details the integration of Milvus as a core component in an intelligent wardrobe customization system. It explains how Milvus is used for vector similarity search to match user requirements with wardrobe components, complementing Elasticsearch for initial filtering and PostgreSQL for structured data. The article outlines the system's architecture, data flow (offline and online), and the specific roles of Elasticsearch and Milvus in the recall and ranking process. It highlights Milvus's suitability for handling complex, unstructured wardrobe data by converting it into vectors for efficient similarity search.

6/23/2022

Details the implementation of video deduplication using Milvus, including the workflow of extracting feature vectors from video frames, indexing these vectors in Milvus, and performing similarity searches to identify duplicate videos. The system architecture involves Kafka for data ingestion, deep learning models for feature extraction, Milvus for vector indexing and search, Ceph for storage, and TiDB/Pika for video ID mapping. The similarity search process is described in three steps: batch recall, refined search based on video IDs, and final similarity scoring.

6/21/2022

This post introduces the fundamental concepts of vector databases and similarity search, explaining how unstructured data is represented as vectors (embeddings) and the challenges of storing, representing, and querying this data. It details the necessity of vector databases for handling massive amounts of vectors efficiently, outlining key requirements such as scalability, reliability, and speed. The post then introduces Milvus as an open-source vector database, highlighting its role in accelerating MLOps and its capabilities in storing, indexing, and querying embedding vectors, including support for attribute filtering using a bitmask mechanism.

6/20/2022

This post details the implementation of Zhentu, a photo fraud detection system, using Milvus. It describes the system architecture involving image pre-processing, feature vector extraction using deep learning, normalization, and vector search with Milvus. The deployment section outlines using Milvus cluster on Kubernetes with Helm for high availability and real-time synchronization. It also provides specific configurations for index building (IVF_PQ with nlist=2048) and search parameters (IP metric_type with nprobe=32), highlighting the performance benefits of using Inner Product over Euclidean distance for normalized vectors. The post reports real-world performance metrics, including detecting over 20,000 fake licenses and achieving sub-second query times for tens of millions of vectors.

5/16/2022

This post analyzes the limitations of consensus-based replication (Raft, Paxos) in cloud-native databases, particularly concerning availability, complexity, and performance. It proposes an alternative log replication strategy for cloud-native and distributed databases based on two principles: 1) Replication as a service, decoupling synchronization from storage, and 2) The 'Russian doll principle', leveraging external services for logging (e.g., Kafka/Kinesis) and storage (e.g., S3), and using coordination services (e.g., etcd) for leader election. The post also touches upon Milvus's tunable consistency features, such as follower reads, as a means to improve cluster throughput and reduce leader load.

5/10/2022

This post introduces Knowhere, Milvus's core vector computation engine. It details Knowhere's concept, its role within the Milvus architecture, and its advantages over Faiss, including support for BitsetView, more similarity metrics for binary vectors, and AVX512 instruction sets with automatic SIMD instruction selection. The post also breaks down the Knowhere code structure, explaining base classes and various index types (IDMAP, IVF, Annoy, HNSW), and provides a guide on how to add new indexes to Knowhere.

5/5/2022

This post details the internal workings of Milvus's query execution engine. It explains how query expressions with scalar filtering are parsed using EBNF syntax, how an Abstract Syntax Tree (AST) is generated using ANTLR and the Expr package, and how the query is executed through a visitor design pattern on protobuf structures, resulting in a bitmask output. It also provides an example of a complex film query and its step-by-step processing.

4/6/2022

This post details the data insertion workflow in Milvus, explaining the roles of proxy, data coord, data node, and root coord. It elaborates on data organization (collection, partition, shard/channel, segment) and the different states of segments (growing, sealed, flushed). It also describes data persistence through binlog files and the mechanisms for segment allocation and channel assignment (consistent hashing, load balancing). The Time Tick mechanism for timestamping and data consumption is also explained.

3/14/2022

This post details the thinking process and design principles behind building Milvus, an open-source, cloud-native vector database for scalable similarity search. It explains the Milvus architecture in detail, covering the need for a basic software stack for unstructured data, the conversion of unstructured data to vectors, and the role of similarity metrics. It differentiates Milvus as a vector database from a vector search engine by highlighting features like sharding, partitioning, replication, disaster recovery, load balancing, and query parsing. The cloud-native approach is discussed, including the separation of compute and storage ('shared something' architecture) and the use of Kubernetes for microservices. The design principles of Milvus 2.0 are elaborated: 'Log as data' where logs record all changes and worker nodes consume them; 'Duality of table and log' where logs can be converted to tables and aggregated into segments; and 'Log persistency' using a publish-subscribe system (log broker) for decoupling and statelessness. The deployment modes (standalone and cluster) are described, along with a high-level overview of the Milvus architecture's four layers: access, coordination, distributed execution, and storage.

2/21/2022

This post details the concept and implementation of data compaction in Milvus 2.0. It explains that compaction merges small segments into large ones and cleans logically deleted data to save storage space and improve query efficiency. It covers configuration parameters (`dataCoord.enableCompaction`, `common.retentionDuration`), manual and automatic triggering mechanisms, and the specific processes for binlog compaction (based on delta binlog row count or size) and segment compaction (based on the number of small sealed segments).

2/14/2022

This post details the internal mechanism of bitsets in Milvus, explaining how they are used to support Time Travel, attribute filtering, and data deletion. It provides concrete examples of bitset generation and manipulation based on DML events and query timestamps, illustrating the logical operations involved in determining which entities are included in search or query results.

2/7/2022

This post details the implementation of the DELETE function in Milvus 2.0, enabling logical deletion of entities. It explains how data nodes handle INSERT and DELETE messages, including the use of bloom filters and Delta binlogs. It also describes how query nodes process these messages, distinguishing between growing and sealed segments, and how deleted entities are omitted during search operations using bitsets. The post also touches upon the impact of consistency levels on the visibility of deleted data.

1/20/2022

Introduced the concept and implementation of collection aliases as a mechanism for hot reloading data in Milvus. This allows for dynamic updates of large datasets by swapping aliases to point to new collections, avoiding the need for data deletion/insertion or client code changes. The implementation was contributed by SmartNews and officially released with Milvus 2.0.0-RC7.

12/21/2021

This post details the design thinking behind Milvus 2.0's cloud-scalable cluster architecture. It contrasts with traditional shared-everything clusters, highlighting the challenges of global locking and buffer cross-invalidation in cloud environments. Milvus 2.0 adopts an eventual consistency model and asynchronous processes, utilizing cloud messaging queues (Pulsar/Kafka) for log storage and data nodes to bridge local storage with cloud shared storage (S3). The architecture aims for high scalability and availability by moving away from centralized components and embracing a more distributed, event-driven approach. Data is managed in collections with physical segments, and partitioning is discussed as a strategy for performance.

12/10/2021

This post details the construction of a trademark similarity search system using Milvus. It outlines a four-step process: preparing a logo dataset, training an image feature extraction model (specifically VGG16), converting logos into vectors, and storing/searching these vectors in Milvus. The post emphasizes the use of Milvus's vector similarity search capabilities and suggests organizing data into collections based on the International Classification of Goods and Services for improved search performance.

11/26/2021

This post details the integration of Milvus with PaddlePaddle for building a product recommender system. It explains the candidate generation and ranking stages of recommender systems, the architecture involving MIND, PaddleRec, and Milvus, and the implementation steps including data processing, model training, and generating product item candidates. Specifically, it describes how item vectors are stored in Milvus and how user vectors are used for similarity search and ranking.

11/8/2021

This post introduces Milvus as an AI-oriented general-purpose vector database system designed to handle unstructured data. It discusses the challenges of traditional data analytics with unstructured data and the emergence of the vector data layer through embedding technology. The post details the major attributes of a vector database, including support for high-efficiency vector operators, vector indexing, consistent user experience, hybrid search, and cloud-native architecture. It then presents the Milvus 2.0 architecture, emphasizing 'log as data', unified batch and stream processing, stateless components, and microservices, with detailed explanations of data persistence through metadata, log, and object storage, and the role of the log broker in ensuring data reliability and decoupling components.

10/11/2021

This post details the construction of a video analysis system leveraging Milvus. It outlines the process of extracting frames from videos, performing object detection using YOLOv3, extracting feature vectors from detected objects with ResNet-50, and finally utilizing Milvus for vector similarity search to analyze video content. Code snippets demonstrate frame extraction with OpenCV, object detection, feature vector extraction, and Milvus search operations.

9/26/2021

This post details a method for performing image search by combining multiple AI models (VGG16, ResNet50) using ONNX for feature vector generation and Milvus for vector retrieval. It provides Python code snippets for converting Keras models to ONNX format, extracting feature vectors using ONNX runtime, storing these vectors in Milvus collections, and performing similarity searches using Euclidean distance. It also mentions the use of Redis for mapping vector IDs to image paths in the context of Milvus 1.1.

9/24/2021

This post introduces and explains DiskANN, a disk-based ANNS solution inspired by the NeurIPS 2019 paper. It details the motivations for such a solution, highlighting the limitations of memory-resident graph-based indexing for large datasets. The post describes the Vamana algorithm, a novel graph-based approach with a flexible trimming strategy, and how DiskANN adapts it for SSD-resident indexing. Key optimizations like caching hotspots and beam search are explained to reduce disk I/O. The post also includes experimental comparisons of Vamana with NSG and HNSW, and outlines the experimental setup for DiskANN.

9/6/2021

This post details the application of Milvus for DNA sequence classification. It explains the k-mer preprocessing technique to convert DNA sequences into manageable text-like data, followed by vectorization using NLP models like CountVectorizer. The post demonstrates how Milvus's similarity search capabilities, based on ANN, can be used to classify gene families and determine genetic similarity between organisms. It includes code snippets for inserting vectors and performing searches, and presents performance comparisons showing Milvus outperforming MySQL and other ML classifiers in accuracy for DNA sequence classification.

8/26/2021

This post analyzes the HM-ANN algorithm, a novel approach for Approximate Nearest Neighbor (ANN) search that addresses memory heterogeneity by optimizing index placement across DRAM and Persistent Memory (PMem). It details HM-ANN's graph construction (top-down insertion, bottom-up promotion with high-degree node selection) and search algorithms (fast memory search, parallel layer-0 search with prefetching). The evaluation compares HM-ANN against other ANN algorithms like HNSW and Disk-ANN on billion-scale datasets, highlighting its performance benefits in terms of latency and recall, especially when datasets exceed DRAM capacity. It also discusses the effectiveness of high-degree promotion and the performance gains from memory management techniques like data prefetching.

8/1/2021

Milvus 2.0 is a refactored cloud-native vector database built on principles of cloud-native first, logs as data, and unified batch and stream processing. It features a microservice architecture with storage and computing separation, read/write separation, and incremental/historical data separation. The system architecture includes an access layer, coordinator service (root coord, data coord, query coord, index coord), worker nodes (data nodes, query nodes, index nodes), and storage (etcd for meta, Pulsar for log broker, MinIO/S3 for object storage). Key features introduced include always online capabilities through "fail cheap, fail small, fail often" strategies, hybrid search between scalar and vector data, tunable consistency, time travel for data rollback, and an ORM Python SDK. Supporting tools like Milvus Insight (GUI) and out-of-box deployment via helm/docker-compose are also highlighted.

7/29/2021

This post details Vipshop's implementation of a personalized product recommender system using Milvus. It describes a read/write separation architecture, the use of Redis for managing collection aliases to enable seamless data updates (effectively near real-time updates by switching entire data collections), batch writing data to multiple partitions, index building and pre-loading for performance, and a data recall process that achieves an average latency of 30ms. It also highlights lessons learned regarding connection pooling for the Java client, index warm-up, and parameter tuning (nlist, nprobe).

7/27/2021

This post details the application of Milvus for audio retrieval. It explains how audio data is processed by extracting feature vectors using deep learning models (specifically PANNs) and then how Milvus is used for vector similarity search on these extracted features. The post outlines the system overview with insert and search steps, including code snippets for embedding generation and insertion into Milvus, and subsequent search operations. It also highlights the use of a MySQL database for correlating Milvus IDs with audio file names.

7/20/2021

This post details the architecture and implementation of MilMil, a Milvus-powered FAQ chatbot. It explains how Milvus is used for vector similarity retrieval to answer user questions. The system utilizes the `sentence-transformers/paraphrase-mpnet-base-v2` model for generating vector representations of FAQs, which are then stored in Milvus. PostgreSQL is used to store question-answer pairs linked by vector IDs. When a user submits a question, it's converted to a vector, and Milvus retrieves the most semantically similar vectors. The corresponding questions and answers are then presented to the user. The post also provides a link to the relevant code in the Milvus bootcamp.

7/9/2021

This post details the implementation of an image similarity search system for the Stylepedia app using Milvus. It covers the system overview, breaking down the implementation into garment detection (YOLOv5), feature extraction (EfficientNet with deep metric learning), vector similarity search (Milvus with HNSW index and partitioning), and post-processing. The system vectorizes images into 512-dimensional feature vectors and uses Milvus for near real-time search on millions of vectors, enabling features like user uploads, outfit suggestions, and fashion photograph recommendations.

6/15/2021

This post details the integration of Milvus with JuiceFS, a cloud-native distributed POSIX file system. It explains how to configure Milvus to use JuiceFS for shared file storage, including setting up MySQL for metadata, creating a JuiceFS file system on object storage (e.g., Azure Blob Storage) and Redis for metadata, and configuring Milvus nodes (read/write roles) and Mishards. Performance benchmarks are presented, highlighting JuiceFS's advantages over EFS in IOPS and throughput, and demonstrating near-instantaneous data loading from disk to memory.

6/8/2021

This post details how Sohu News integrated Milvus into their news recommendation system. It explains the use of semantic vector search with BERT-as-service for encoding news articles and user profiles, the workflow for generating user semantic vectors daily, and the insertion of these vectors into Milvus. A key technical contribution is the optimization strategy using MinHash and Jaccard distance to reduce semantic vector extraction time from days to hours. Additionally, it describes a method for resolving short text classification errors by querying Milvus with short text vectors to find similar long text articles and using a voting mechanism for reclassification. The post highlights Milvus's role in improving real-time performance and the efficiency of text classification.

5/19/2021

This post details how Milvus, an open-source vector database, can be applied in the financial services industry to accelerate AI initiatives. It outlines specific use cases including enhancing customer experience with banking chatbots, boosting sales with recommender systems, and analyzing financial data through semantic text mining. The post emphasizes Milvus's capability to manage and search massive vector datasets, enabling similarity searches for AI applications.

4/29/2021

This post introduces Milvus v1.0, detailing its core features and optimizations. It covers CPU/GPU distributions, vector entity structure with integer IDs, partitioning for search acceleration, and enhancements to index algorithms (IVF, FLAT, IVF_SQ8H). Performance optimizations for search (nq < threads, request combining) and index building are highlighted. It also mentions distance metrics, logging configuration, the Mishards distributed solution, Prometheus integration for monitoring, and metadata management (SQLite/MySQL).

4/23/2021

This post details Trend Micro's implementation of Milvus for real-time Android virus detection. It describes their system architecture, which involves converting APK Thash values into binary vectors and storing them in Milvus for similarity search. The post highlights the use of Hamming distance and specific index types (FLAT, IVF_FLAT) for binary vectors. It also discusses the integration with MySQL and Redis for storing Sha256 values and caching, respectively. The deployment utilizes Mishards for sharding and Kubernetes for high availability. Monitoring with Prometheus and Grafana is also covered.

4/19/2021

This post details how Milvus, leveraging FAISS, can be used as a vector search engine to achieve high-performance semantic search, demonstrating its capability to handle millions of vectors and thousands of requests per second with low latency. It highlights the integration of Milvus with other systems like Solr for a complete semantic search workflow and discusses the performance implications of different index types (FLAT vs. HNSW) and scale testing results.

4/8/2021

This post demonstrates the practical application of Milvus in building four distinct AI applications: NLP chatbots, reverse image search, audio search, and video object detection. It outlines the specific ML models (BERT, VGG, PANNs, YOLOv3, ResNet50) used in conjunction with Milvus for each use case, detailing the data processing steps (vectorization and storage) and search mechanisms. For NLP, it explains how BERT converts questions to vectors, and Milvus retrieves similar questions, with answers stored in PostgreSQL. For image search, VGG converts images to vectors, Milvus finds similar vectors, and image paths are stored in CacheDB. For audio search, PANNs convert audio to vectors, Milvus finds similar vectors using IP distance, and audio paths are stored in PostgreSQL. For video object detection, ResNet50 converts object images to vectors, Milvus finds similar objects, and image paths are stored in MySQL, with OpenCV and YOLOv3 used for video processing.

3/31/2021

This post introduces Milvus as a vector data management platform designed to support the transition to Software 2.0 and operationalize AI at scale with MLOps. It explains MLOps principles (CI/CD, MDE, champion-challenger testing, model versioning, model store/rollback) and highlights how Milvus facilitates these by providing an environment for model training, cross-team collaboration, model governance, comprehensive APIs for popular frameworks, compatibility with Google Colaboratory, and AutoML functionality. It positions Milvus as a solution for managing trillion-scale vector datasets and powering vector similarity search for unstructured data.

3/18/2021

This post details how Beike used Milvus to build an AI-infused proptech platform for personalized real estate search. It explains the concept of vector similarity search, how Milvus manages vector data, and Beike's specific implementation. Beike converts property listings into feature vectors, indexes them in Milvus, and uses similarity search (cosine distance) to recommend similar homes based on user preferences and listing characteristics. The system achieves an average query time of 113ms on 3 million vectors and uses an A/B table switching mechanism for data updates.

3/16/2021

This post details iYUNDONG's implementation of an intelligent image retrieval system using Milvus to power their sports app. It highlights the challenges of handling massive datasets, achieving instant search availability, and efficient feature extraction. Milvus was chosen for its ability to perform fast vector retrieval on ultra-large datasets and mass storage at lower costs, enabling millisecond-level queries on over 60 million images with an accuracy rate above 92%. The system architecture involves converting facial features into 128-dimensional vectors and storing them in Milvus, with additional metadata stored in a third-party database, linked via the Leaf algorithm for distributed ID generation.

3/10/2021

Milvus v1.0 release introduces support for a wider range of similarity metrics, improved integration with state-of-the-art ANNs algorithms, and enhanced scale-out capabilities through the Mishards sharding proxy. It also adds support for various AI processors like X86, Nvidia GPU, and Xilinx FPGA. The release signifies a commitment to long-term support for v1.0 and a focus on developing toolchain enhancements and fostering AI ecosystem integration. Future development will utilize Golang for the Milvus engine and C++ for ANNS algorithms, with the distributed version aiming to utilize existing cloud components.

2/25/2021

This post details the integration of Milvus with PaddlePaddle to build a personalized recommender system. It outlines the data preparation using the MovieLens Million Dataset, the Fusion Recommendation Model implemented in PaddlePaddle, and the system architecture where Milvus stores movie feature vectors and performs similarity searches based on user features. It provides steps for model training, data preprocessing, and implementing the recommender system with Milvus, including example code and output for movie recommendations.

2/5/2021

This post details Tokopedia's implementation of semantic search using Milvus. It covers their transition from keyword-based search with Elasticsearch to vector representation and similarity search. The decision to use Milvus was based on its ease of use and comprehensive index support compared to FAISS. The post also describes the deployment of a Milvus cluster with Mishards for high availability, and explains the fundamental concept of vector indexing for accelerating similarity search.

1/13/2021

This post details Milvus's architecture for massive-scale vector similarity search, focusing on its use of LSM trees for efficient dynamic data management, optimized data handling with WAL and buffer flushing, segment merging to reduce fragmentation, and the acceleration of similarity search through ANN indexing. It also mentions the use of OLTP systems (SQLite/MySQL) for metadata management.

1/13/2021

This post introduces Milvus as an open-source data management platform for massive-scale vector data and MLOps. It highlights the use of AI to convert unstructured data into vectors and the necessity of ANN algorithms for efficient similarity search. Milvus integrates libraries like Faiss, NMSLIB, and Annoy to provide a unified platform for storing, processing, and analyzing large vector datasets. It supports CPU/GPU computing, distributed architecture for scalability, and compatibility with various AI models, programming languages, and processor types.

1/5/2021

This post provides a high-level overview of vector similarity search, explaining what it is, its applications (e-commerce, security, recommendation engines, chatbots, image/video search, chemical similarity search), and introduces Milvus as an open-source vector data management platform. It discusses the conversion of unstructured data to feature vectors using neural networks and the use of Approximate Nearest Neighbor (ANN) algorithms to accelerate similarity search.

12/1/2020

This post details the integration of Milvus as a vector database within a graph-based recommendation system. It outlines the process of using PinSage and DGL to generate movie embeddings from the MovieLens dataset, loading these embeddings into Milvus for storage and indexing, and then performing vector similarity searches to retrieve similar movie information. The post also touches upon data preprocessing and the general workflow of building such a system.

11/11/2020

This post introduces Milvus as a vector data management platform capable of building large-scale unstructured data analytics systems, specifically highlighting its ability to handle 1+ billion images for reverse image search. It details how AI enables unstructured data analytics by converting data into feature vectors, discusses the limitations of existing approaches (flexibility, dynamic data handling, advanced query processing, heterogeneous computing), and positions Milvus as a solution addressing these challenges. Milvus supports multiple SDKs, index types, and provides optimizations for CPUs and GPUs, achieving high recall rates and fast query times on massive datasets.

9/11/2020

This post describes how the Cleveland Museum of Art's Digital Innovation team used Milvus, an open-source vector similarity engine, to build the ArtLens AI: Share Your View tool. The tool uses AI and machine learning to recognize shapes, patterns, and objects in user photos and find matching artworks from the museum's collection. This demonstrates a practical application of Milvus for reverse image search and content matching in a cultural heritage context.

9/8/2020

This post details the implementation of an item-based collaborative filtering (I2I CF) recommender system for music, leveraging Milvus as the embedding similarity search engine. It describes the process of extracting song feature embeddings using CNNs from Mel-frequency cepstrum (MFC) and deploying Milvus in a highly available cluster configuration with Mishards on Kubernetes for production. Additionally, Milvus is used as a preliminary step for duplicate song filtering by searching for similar audio fingerprints.

8/29/2020

This post details the architecture and implementation of a video search system using Milvus 0.7.1. It outlines the process of extracting key frames from videos, generating embeddings using the VGG model, and storing these embeddings in Milvus. The system also utilizes Minio for video storage and Redis for mapping correlations between videos and vectors. Deployment is managed via Docker and docker-compose, with a specific script for importing video data.

8/22/2020

Introduced a soft deletion mechanism for flushed data by saving deleted IDs in a separate DEL file, which are filtered during read operations. Implemented segments as the basic unit for data operations, containing raw files, index files, and a DEL file. Utilized counting bloom filters per segment to optimize the identification of segments containing deleted IDs.

8/12/2020

This post details the technical implementation of a search-by-image system using CNNs (VGG16) for feature extraction and Milvus as the vector search engine. It covers feature extraction using Keras/TensorFlow, image preprocessing (normalization, bytes conversion, black border removal), Milvus requirements (CPU instruction set), capacity planning for billion-scale datasets (calculating storage for 512-dimensional float32 vectors), system configuration, database design considerations (collections, partitions, metadata management with SQLite/MySQL, limitations on number of collections/partitions), structured data and vector mapping via IDs, index selection, and processing search results (ID + distance, filtering -1 IDs, pagination).

8/4/2020

This post details the first-generation image search system built by Yupoo Picture Manager, which used the pHash algorithm for feature extraction and Elasticsearch for similarity calculation based on Hamming distance. It explains the limitations of pHash, particularly its inability to handle image integrity destruction, and sets the stage for a second-generation system that would likely employ more advanced vector embedding techniques for billion-scale image search.

7/28/2020

This post details the integration of Milvus into WPS Office's AI-powered writing assistant. It describes the process of extracting features from unstructured text using TFIDF and a bi-directional LSTM-CNNs-CRF deep learning model, creating sentence embeddings with Infersent, and then storing and querying these embeddings in Milvus. Specifically, it highlights the use of the IVF_FLAT index and Milvus's partitioning function to improve query performance for sentence embeddings, achieving average query times of 0.2 seconds.

6/4/2020

This post details Xiaomi's implementation of an AI-powered news recommendation engine within their mobile browser, leveraging Milvus as the core data management platform for vector similarity search. It explains the two-stage recommendation process (retrieval and sorting) and highlights the integration of Milvus with BERT (specifically SimBert) for vectorizing article titles. The system uses Milvus to store and query vector embeddings of news articles, enabling efficient similarity search. The post also touches upon data update strategies (full and incremental) and the importance of Milvus's ability to rapidly incorporate and search new data for real-time recommendations.

5/13/2020

This post details the architecture and steps for building an intelligent QA system using Milvus and the BERT NLP model. It describes how to convert questions into feature vectors using BERT, store these vectors in Milvus for similarity search, and use PostgreSQL to retrieve corresponding answers based on the vector IDs. The system architecture, data preparation, feature vector generation, and answer retrieval process are explained with diagrams and code snippets.

4/13/2020

This post details Milvus's internal mechanism for handling dynamic data updates and near real-time queries. It introduces the MemTable and MemTableFile architecture for managing data in memory, dividing MemTableFiles into mutable and immutable states. Data is inserted into a mutable MemTableFile, which transitions to an immutable state and is then serialized to disk by a background thread. This process ensures data is available for searching shortly after insertion, with a maximum latency of one second. The post also touches upon merging serialized files and asynchronous index building.

3/17/2020

Introduces Mishards, a distributed solution for Milvus enabling horizontal scaling for massive datasets (10B+ vectors). Details the Mishards architecture including service discovery (ZooKeeper, etcd, Consul), load balancing (Nginx, HAProxy), Mishards nodes (stateless, scalable), write-only Milvus nodes (single, HA required), read-only Milvus nodes (stateful, scalable), shared storage, and metadata service (MySQL). Explains how Mishards breaks down requests, routes them to read-only nodes, and merges results. Discusses read-only node scalability through sharding and horizontal scaling. Highlights the integration with OpenTracing for service invocation tracing and Prometheus/Grafana for monitoring.

2/26/2020

This post provides best practices for selecting index parameters for the IVF index type in Milvus. It details the impact of `index_file_size` on search performance and memory loading, and explains the roles of `nlist` (number of clusters) and `nprobe` (number of clusters to search) in balancing search performance, precision, and computation load. It includes experimental results and recommendations for setting these parameters.

2/6/2020

This post details the integration of Milvus with RDKit for high-performance chemical structure similarity search in drug discovery. It explains how RDKit generates Morgan fingerprints from chemical structures (SMILES format) and how these fingerprints are imported into Milvus for substructure, similarity, and exact searches. The post highlights the use of Tanimoto distance (implemented as Jaccard distance in Milvus) for chemical fingerprint computation and demonstrates a system demo with over 90 million chemical fingerprints.

12/31/2019

This post details the fields within the `Tables` and `TableFiles` metadata tables in Milvus versions 0.5.0 and 0.6.0. It explains the significance of fields like `index_file_size`, `engine_type`, `nlist`, `metric_type`, `state`, `created_on`, `owner_table`, `partition_tag`, and `version`. It also describes fields in `TableFiles` such as `file_type`, `file_size`, and the number of vector rows, and clarifies the relationship between data files and their corresponding index files. The post also highlights the evolution of the data file cleanup mechanism in v0.6.0, moving from a fixed interval hard-delete to a reference counting approach for improved reliability and reduced disk usage.

12/27/2019

This post details the fields within Milvus's metadata tables (`Tables` and `TableFiles`) for versions 0.5.0 and 0.6.0. It explains the purpose and data types of fields related to table configuration (e.g., dimension, index type, metric type), state, and file management (e.g., file size, row count, file type). It also introduces new fields added in 0.6.0 for partition management, such as `owner_table` and `partition_tag`.

12/25/2019

This post details how to view Milvus metadata by configuring and querying either SQLite or MySQL as the metadata backend. It provides specific commands for setting up SQLite and MySQL clients and for querying the `Tables` and `TableFiles` tables.

12/24/2019

This post details how to view Milvus metadata by connecting to either SQLite or MySQL databases. It provides specific command-line instructions for installing clients and querying the `Tables` and `TableFiles` tables, demonstrating how to inspect the internal structure of Milvus's data management.

12/5/2019

This post details the role of vector indexing in accelerating similarity search, focusing on different Vector Inverted File (IVF) index types: FLAT, IVF_FLAT, IVF_SQ8, and IVF_SQ8H. It provides performance test results for query time and recall rates for each index type in Milvus, offering guidance on selecting the best index for various scenarios based on dataset size, accuracy requirements, and resource availability. It also introduces the concept of scalar quantization in IVF_SQ8 for reduced resource consumption.

11/8/2019

This post details Milvus's data management strategies for massive-scale vector search. It explains vector insertion using a mutable buffer that flushes to disk regularly, similar to Elasticsearch. It describes the creation and merging of Raw Data Files to manage fragmentation and the process of building Index Files for efficient search, contrasting them with raw data. The post also covers Meta Data management using SQLite/MySQL to track file statuses and operations, and introduces the Query Scheduler for optimizing hardware resource utilization during searches, including 'cold' vs. 'warm' queries and LRU for data replacement. Finally, it touches upon the Result Reducer for combining search results.