Data Addressing in Storage Systems

1 post · Aug 2021 – Mar 2026

What this trail covers

Milvus continues to evolve its storage and indexing strategies to reduce memory costs and improve performance at scale. Building on DiskANN, AISAQ further minimizes DRAM footprint by storing all search-critical data on disk and optimizing data placement to minimize I/O operations. This enables billion-scale vector search with significantly lower memory usage while maintaining practical performance. DiskANN is available as the DISKANN index type, leveraging SSDs for cost-effective, large-scale vector search.

All Milvus trails

Mar 2026A Deep Dive into Data Addressing in Storage Systems: From HashMap to HDFS, Kafka, Milvus, and IcebergThis post details how Milvus, like other large-scale storage systems (HDFS, Kafka, Iceberg), optimizes data addressing to reduce latency. It explains that the bottleneck is rarely bandwidth or compute, but rather the process of translating logical identifiers to physical data locations. The post categorizes optimizations into three core techniques: Computation (deriving location directly), Caching (keeping metadata/indexes in memory), and Pruning (ruling out irrelevant data). It uses the 'Two Sum' problem as an analogy to illustrate the shift from brute-force search to direct addressing via computation, and then applies these principles to HashMap, HDFS, and Kafka before discussing Milvus and Iceberg in the context of object storage. For Milvus specifically, it highlights how vector indexes (like HM-ANN mentioned in the context of data addressing) are a form of computation and caching to locate embeddings efficiently, reducing I/O operations.
Mar 2026How to Cut Vector Database Costs by Up to 80%: A Practical Milvus Optimization GuideThis post details practical optimization strategies for reducing vector database costs, primarily focusing on memory optimization in Milvus. It quantifies cost components, highlighting memory as the dominant factor (85-90%). It provides a detailed comparison of index types (FLAT, IVF_FLAT, HNSW, IVF_SQ8, IVF_PQ, DiskANN) and their memory multipliers, query speeds, and recall, recommending IVF_SQ8 for cost-sensitive production workloads. The post introduces and explains MMap (Memory-Mapped Files) and tiered storage as methods to reduce memory usage by 60-80%, detailing their data flow, resource usage, latency characteristics, and hardware recommendations (NVMe SSDs). Configuration instructions for both MMap (YAML and Python SDK) and tiered storage (YAML) are provided. Finally, it emphasizes the cost impact of embedding dimensionality, recommending lower-dimensional embeddings for significant infrastructure savings.
Dec 2025Introducing AISAQ in Milvus: Billion-Scale Vector Search Just Got 3,200× Cheaper on MemoryIntroduces AISAQ (Approximate Indexing for Storage-Aware Querying), a disk-based vector index that eliminates the need for PQ data in DRAM. AISAQ reorganizes node storage so that full vectors, neighbor lists, and PQ information are arranged on disk in patterns optimized for access locality, reducing memory usage from 32 GB to about 10 MB for a billion-vector workload.
May 2025DiskANN ExplainedThis post provides a technical deep dive into DiskANN, a graph-based vector search method that leverages SSDs to store the index, significantly reducing memory requirements. It explains the Vamana algorithm for graph construction, which uses a divide-and-conquer strategy to reduce memory usage. The post details how DiskANN optimizes disk access by storing the global search graph and full-precision vector embeddings on SSD, using fixed-size data structures for node addressing and fetching multiple nodes per read request. It also describes the use of product quantization in RAM for approximate similarity calculations to guide SSD reads, ensuring higher recall by performing final searches with full embeddings retrieved from SSD. The article also mentions FreshDiskANN for index updates and provides Python code examples for using the DISKANN index type in Milvus.
Nov 2023Milvus Introduced MMap for Redefined Data Management and Increased Storage CapabilityThis post introduces the MMap (memory-mapped files) feature in Milvus 2.3, which allows large files to be mapped directly into the system's memory space. This eliminates explicit read/write operations, enabling efficient storage and random access for large files. The feature is recommended for users less sensitive to performance fluctuations, as it allows processing more data within limited memory. MMap optimizes data access by writing scalar data directly to disk for sequentially accessed segments and using an offsets array in memory for variable-length data. Vector indexes selectively employ MMap while retaining adjacency lists in memory. Internal testing shows Milvus can handle double the data volume with MMap enabled. The feature is enabled by modifying the `milvus.yaml` file by adding `mmapDirPath` under the `queryNode` configuration.
Advertisement
Sep 2021DiskANN, A Disk-based ANNS Solution with High Recall and High QPS on Billion-scale DatasetThis post details the DiskANN algorithm, a disk-based Approximate Nearest Neighbor Search (ANNS) solution designed for billion-scale datasets on a single machine with limited RAM. It introduces the Vamana graph-based algorithm, which minimizes disk access by controlling the search radius and trimming strategy. DiskANN addresses the challenges of indexing and searching large datasets with limited memory by dividing data into clusters, building memory-based Vamana indexes for each, and merging them. It also proposes solutions for distance calculation with compressed vectors and optimizing disk access through caching hotspots and beam search.
Aug 2021Paper Reading｜HM-ANN When ANNS Meets Heterogeneous MemoryThis post introduces HM-ANN, a novel graph-based Approximate Nearest Neighbor (ANN) search algorithm designed for heterogeneous memory environments. HM-ANN addresses the trade-off between query accuracy and latency caused by limited DRAM capacity by intelligently partitioning graph layers between fast DRAM and slower but larger Persistent Memory (PMem). It employs a high-degree promotion strategy during graph construction and a two-phase search algorithm (fast memory search and parallel layer-0 search with prefetching) to optimize performance and accuracy for billion-scale datasets.