OIL+VCache: File abstraction for distributed systems

What this post added

Introduced OIL (Output-Input Language) as a unified namespace and abstraction for files across heterogeneous storage systems, allowing configuration of I/O trade-offs via Directed Acyclic Graphs (DAGs) instead of coding. Coupled OIL with VCache, a distributed virtual memory system, to provide significant efficiency and performance gains. OIL extends POSIX APIs to better capture distributed system nuances, enabling per-file or per-I/O trade-off adjustments and allowing files to retain the same name even when migrating storage systems. VCache integrates with OIL to provide layered caching, supporting various write modes and delegation for buffering and durability. The system uses DAGs for both data and metadata, with the metadata DAG handling mutual exclusion and locking, and the data DAG managing byte-stream operations. OIL+VCache has been deployed to enhance live video viewer systems, improving reliability and latency while reducing computational overhead.