TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

Redis Latency Diagnosis and Mitigation

1 post · Nov 2020 – Nov 2022

What this trail covers

GitLab is continuously optimizing platform performance and reliability. This includes implementing rate limiting for unauthenticated users of the Projects List API to mitigate server load and prevent abuse. Additionally, security measures are being enhanced to combat increased volumes of password attacks against the OAuth API endpoint, with recommendations for users to enable two-factor authentication and leverage features like Restrict Group Access by IP Address and Git Abuse Rate Limiting. New efforts focus on diagnosing and mitigating Redis latency spikes through detailed analysis of eviction bursts and CPU saturation, aiming to ensure Redis performance and reliability.

All GitLab trails
  1. How we diagnosed and resolved Redis latency spikes with BPF and other toolsThis post introduces a detailed methodology for diagnosing and resolving Redis latency spikes. It showcases the use of BPF and profiling tools (perf, flamescope, funclatency) to analyze Redis behavior during eviction bursts. Key findings include the identification of CPU saturation on the main thread caused by `performEvictions` and the discovery of a three-phase cycle leading to latency spikes. The post also discusses the limitations of Redis 6.2's eviction throttling mechanism for this specific problem.
  2. Lessons in iteration from a new team in infrastructureThis post details lessons learned by the Scalability Team in their project to investigate and improve Redis usage. Key takeaways include the importance of focusing on impact and delivering value iteratively, applying Minimum Viable Change (MVC) principles to scaling problems, and shaping projects to deliver impact throughout the process rather than at the end. The team learned to slice projects differently to achieve impact sooner and re-assess more frequently.