Querying Big Data in Real-time with Presto & Grab's TalariaDB

This post details the design and implementation of TalariaDB, a distributed, highly available, and low-latency time-series database for storing real-time data like logs, metrics, and click streams. TalariaDB addresses the need to query terabytes of data per hour with predictable low latency and cost, storing data in pre-partitioned batches in a binary, columnar format. It leverages the Badger key-value store and integrates with Presto via the Presto Thrift Connector for SQL querying. The system focuses on simplicity, linear horizontal scalability, low latency, availability, zero operation, high write throughput, and cost-effectiveness.