TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

Bitset for Data Management

1 post · Aug 2020 – Feb 2022

What this trail covers

Milvus 2.0 introduces bitsets as a core mechanism to support Time Travel, attribute filtering, and data deletion. This enables efficient querying of historical data states, precise filtering based on metadata, and robust handling of deleted records within the vector database. The delete function in v0.7.0 introduced soft deletion for flushed data, saving deleted IDs in a separate DEL file and using a bitmap to filter them during read operations. Segments were introduced as the basic unit for managing raw files, index files, and DEL files. Future optimizations include using counting bloom filters for faster segment identification during deletion.

All Milvus trails
  1. How Bitset Enables the Versatility of Vector Similarity SearchThis post details the concept of bitsets in Milvus, explaining their implementation as arrays of bits representing data validity (1 for valid, 0 for invalid). It illustrates how bitsets are used to manage deleted entities, enable Time Travel queries by filtering data at specific timestamps, and support attribute filtering. Three detailed case studies demonstrate the bitset generation and combination logic for these features, showing how the final `result_bitset` determines which entities are included in search or query operations.
  2. How Milvus Realizes the Delete FunctionThis post details the implementation of the delete function in Milvus v0.7.0. It introduces soft deletion for flushed data by saving deleted IDs in a DEL file and using a bitmap to filter them during read operations. It also defines segments as the basic unit for managing raw files, index files, and DEL files. The post discusses the use of counting bloom filters as a performance optimization for identifying segments containing deleted IDs.