Debugging High Latency Due to Context Leaks
Jun 30, 2021Sourabh Suman

Summary
This post details the debugging of high latency in the Market-Store feature store, which was caused by a memory leak in the `grab/async` library. The leak occurred because a recent library update changed the context handling to pass task contexts instead of background contexts to runners, and these task contexts were not being cancelled. This led to uncleaned-up child contexts and a memory leak, eventually causing high latency. The investigation used Go's `pprof` tool to identify the heap profile and pinpoint the memory allocation issue. The solution involved ensuring proper cancellation of contexts using the `CancelFunc` method.