Structured Logging: The Best Friend You’ll Want When Things Go Wrong

This post details the development and adoption of a new structured logging library in Golang to improve observability and reduce MTTR. Key contributions include: 1. Dynamic Log Levels: Enabling runtime adjustment of log levels via configuration management to control verbosity and cost. 2. Consistent Structure in Logs: Implementing the Common Grab Log Schema (CGLS) to enforce deterministic JSON formatting with essential fields for efficient indexing in Elasticsearch, while other fields are placed in a single unstructured field. 3. Plug and Play Support with Grab-Kit: Easy integration with the existing microservice framework. 4. Improved Log Correlation: Automatic inclusion of request IDs and trace IDs to link logs across services. 5. Causal Ordering: Preserving log generation time instead of write time for accurate event sequencing. The library aims to address issues with previous syslog-style logging, high costs, and limitations in leveraging the Elastic stack.