Milvus Introduced MMap for Redefined Data Management and Increased Storage Capability

What this post added

Introduced MMap (memory-mapped files) as a beta feature to redefine data management and increase storage capability. MMap maps large files directly into the system's memory space, transforming files into contiguous memory blocks and eliminating explicit read/write operations. This allows Milvus to process more data within limited memory, especially benefiting users less sensitive to performance fluctuations. Configuration involves adding `mmapDirPath` to `queryNode` in `milvus.yaml`. The feature optimizes data access based on locality, writing scalar data to disk for sequential segments and indexing variable-length data with an offsets array. Vector data uses MMap selectively while retaining adjacency lists in memory. This results in a copy-free streaming process, drastically reducing memory overhead and enabling Milvus to efficiently handle double the data volume.