Application Server Migration (Unicorn to Puma)

2 posts · Jun 2015 – Apr 2021

What this trail covers

GitLab has successfully migrated its application servers from the single-threaded Unicorn to the multi-threaded Puma web server. This migration aimed to address memory growth issues and improve scalability by reducing the number of processes and memory overhead. Puma's multi-threaded model allows for better utilization of CPU and memory, leading to significant memory reduction. The implementation involved extensive testing, tuning of worker/thread configurations, and addressing thread-safety issues. The post also details the use of `unicorn-worker-killer` to manage memory leaks in long-running Unicorn workers by periodically restarting them when memory thresholds are exceeded, ensuring stability and preventing outages.

All GitLab trails

Apr 2021Ruby 2.7: Understand and debug problems with heap compactionThis post details the introduction and debugging of Ruby 2.7's heap compaction feature within Puma 5's `nakayoshi_fork`. It explains the mechanics of compaction, its benefits for memory efficiency, and the challenges encountered when integrating it with C-extensions. A specific production incident caused by Hamlit's improper handling of object marking during compaction is analyzed, leading to the resolution by updating Hamlit to support compaction-aware object marking and pinning.
Jul 2020How we migrated application servers from Unicorn to PumaThis post details the technical migration from Unicorn to Puma web servers. It explains the architectural differences between single-threaded (Unicorn) and multi-threaded (Puma) models, highlighting Puma's benefits for memory reduction (approx. 40%) and scalability. The post outlines the implementation steps, including experimental development use, setting up identical environments for testing, a phased rollout plan, and the resolution of specific issues like thread-safety with ChronicDuration and latency tuning. It also describes an upstream patch to Puma to address uneven capacity distribution. The results show a ~37% drop in memory usage.
Sep 2019Why we created a Memory team at GitLabThis post details the creation of a dedicated Memory team at GitLab to address the application's high memory footprint. It outlines the team's primary goal of reducing the minimum instance memory from 8GB to 1GB. Key initiatives include migrating from the Unicorn web server to Puma, which is expected to reduce memory usage due to its multi-threaded nature. The team is also investigating memory issues in Sidekiq, project imports, and exports, and is working to improve internal development practices related to memory usage and code complexity. The post also mentions the team's current size and ongoing hiring efforts.
Jun 2015How GitLab uses Unicorn and unicorn-worker-killerThis post details GitLab's use of the Unicorn web server and the `unicorn-worker-killer` gem. It explains Unicorn's pre-forking architecture for handling web requests, its multi-process model for CPU utilization and fault tolerance, and how the master process replaces crashed workers. It also describes how `unicorn-worker-killer` monkey-patches Unicorn workers to self-check memory usage after every 16 requests and exit if a pre-set limit is exceeded, allowing the Unicorn master to replace the worker. The post provides log examples of worker timeouts and memory restarts, and mentions tunables for Unicorn like worker count and request timeout.
Advertisement