How We Built a Logging Stack at Grab

This post details the initial build of Grab's logging stack using Elasticsearch, Logstash, and Kibana (ELK Stack). It covers the problem statement of lacking visibility into service logs, the challenges of handling large data volumes (25TB/day), the design decisions for a multi-petabyte cluster including node roles (ingest, coordinator, master, data) and the choice of ingest nodes over Logstash, and the importance of monitoring (Datadog integration). It also highlights pitfalls encountered, such as Field Data Cache exhaustion due to sorting by '_id', the impact of Translog Compression on shard transfers, issues with Segment Memory allocation due to too many shards per node, and the importance of Index Mapping and picking the right Analyser.