Logarithm: A logging engine for AI training workflows and services

What this post added

Introduced Logarithm, a new internal, serverless, multitenant logging engine designed for AI training workflows and services. Logarithm ingests and indexes over 100GB/s of logs in real-time, supporting thousands of queries per second with service-level guarantees on freshness, completeness, durability, and query latency. It supports unstructured text logs with optional metadata key-value pairs, including typed structures and regex-based extraction. The system architecture separates compute (ingestion and query) from storage, utilizes tiered storage (memory, local SSD, blob storage), and employs lightweight, disaggregated secondary indices with Bloom filters. Logarithm's design prioritizes scalability, fault tolerance, and efficient handling of high-throughput, time-ordered log data, with a focus on enabling debugging of complex AI training failures by collocating system and model telemetry.