TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

Data File Cleanup Mechanism

1 post · Dec 2019 – Feb 2022

What this trail covers

Milvus has evolved its data file cleanup mechanism. Previously, files marked for soft-delete were hard-deleted after a fixed 5-minute interval, which was unreliable and led to excessive disk usage. The improved strategy uses a reference counting mechanism where files are hard-deleted only when they are no longer in use by any active task and have been soft-deleted. This ensures reliable deletion and efficient disk usage.

All Milvus trails
  1. How Milvus Deletes Streaming Data in a Distributed ClusterThis post details the implementation of the DELETE function in Milvus 2.0, focusing on logical deletion and its impact on search performance and data recovery. It explains how data nodes handle DELETE messages by generating Delta binlogs and flushing them with sealed segments, and how query nodes process these messages via Delta-Channels to omit deleted entities during searches. The post also highlights the role of bloom filters in identifying segments containing entities to be deleted and the use of bitsets to manage deleted records.
  2. Improvements of the Data File Cleanup MechanismThis post details the previous unreliable 5-minute interval strategy for hard-deleting soft-deleted files in Milvus, which could lead to query failures or excessive disk usage. It introduces the improved strategy implemented in v0.6.0, which uses reference counting to hard-delete files only when they are no longer in use by any task. This ensures reliable deletion and efficient disk usage.