Glean's enterprise AI stack continues to evolve with a focus on optimizing performance and resource costs. This post details the debugging and resolution of a Golang memory leak issue encountered in an App Engine flexible instance. The problem was identified as an interaction between the Golang runtime's memory management (specifically MADV_FREE) and the containerized environment, where memory was not being returned to the OS efficiently, leading to instance termination. The solution involved setting the GODEBUG environment variable to "madvdontneed=1" to revert to MADV_DONTNEED behavior, which resolved the memory leak and stabilized memory usage.
All Glean trails