Context Management and Memory Leak Debugging

2 posts · Mar 2019 – Jun 2021

What this trail covers

Grab has evolved its approach to managing context and deadlines in its microservice architecture to improve resilience and prevent cascading failures. This includes a deep dive into Go's context package, emphasizing the correct configuration of timeouts and deadlines to avoid resource leaks and ensure efficient operation. The post details strategies for setting appropriate timeout values based on SLAs and latency data, and explains how context propagation allows servers to cancel work when deadl

All Grab trails

Jun 2021Debugging High Latency Due to Context LeaksThis post details the debugging of high latency in the Market-Store feature store, which was caused by a memory leak in the `grab/async` library. The leak occurred because a recent library update changed the context handling to pass task contexts instead of background contexts to runners, and these task contexts were not being cancelled. This led to uncleaned-up child contexts and a memory leak, eventually causing high latency. The investigation used Go's `pprof` tool to identify the heap profile and pinpoint the memory allocation issue. The solution involved ensuring proper cancellation of contexts using the `CancelFunc` method.
Mar 2019Context Deadlines and How to Set ThemThis post details a strategy for configuring timeouts and using context deadlines correctly in Go microservices. It explains the fundamental concept of timeouts, their usefulness in preventing cascading failures, and the dangers of poor configuration. It introduces Go's context package for propagating deadlines and cancellation signals, enabling servers to abandon computation when a deadline is exceeded. The article provides guidance on setting effective timeout values based on SLAs and latency data, and discusses the differences between network timeouts, context timeouts, and asynchronous timeouts.
Advertisement