Performance Optimization

6/25/2025

Introduced Exact Code Search, a new capability powered by Zoekt, to provide lightning-fast and precise code search across large codebases. This feature leverages positional trigrams for exact matches, optimized regular expression performance, and a distributed, self-registering node architecture with intelligent load balancing and seamless security integration. It addresses challenges of traditional code search by eliminating false positives and providing contextual multi-line results, significantly improving debugging, code exploration, refactoring, and security auditing workflows.

12/18/2024

This post details how GitLab Premium features can help growing teams scale by addressing common challenges. It highlights specific features for maintaining consistent code quality (Code Quality Reports, Quality Gates, Push Rules), improving collaboration and productivity (Milestones, Epics, Issues), increasing deployment velocity (Merge Trains, Multi-Project Pipeline Graphs, Code Review Analytics), enhancing security and compliance (Protected Branches/Environments, Multiple Approvers, Audit Events), and avoiding downtime (Priority Support).

11/20/2024

This post details how to leverage GitLab Insights Dashboards for data-driven DevSecOps. It explains the various analytics tools available within GitLab (Productivity Analytics, Code Review Analytics, CI/CD Analytics, Value Stream Analytics, Insights) and emphasizes the role of Labels in filtering and focusing metrics. It provides a step-by-step guide on configuring Insights Dashboards using `.gitlab/insights.yml` files for both projects and groups, and offers detailed examples of customizing merge request insights to track squad productivity and requirement types. The post also includes hints on automating label application and provides sample YAML configurations for different dashboard views.

11/12/2024

This post details Kingfisher's adoption and utilization of GitLab for DevSecOps, focusing on CI/CD, automation, and metric tracking (deployment frequency, lead time to change, change failure rates) to improve developer experience and operational efficiency. It highlights their strategic use of GitLab's platform to reduce friction, introduce best practices, and increase automation, with a future focus on AI integration via GitLab Duo.

10/2/2024

This post details the design and evolution of GitLab Reference Architectures, which provide scalable and elastic starting points for deploying GitLab at scale. It outlines the problem of customer deployment hurdles, the goals of performance, availability, scalability, cost-effectiveness, and maintainability, and the iterative process of gathering metrics (using RPS), defining prototype architectures, building, and testing. It highlights the development of the GitLab Performance Tool and GitLab Environment Toolkit to support this process. The post also details the ongoing expansion and refinement of these architectures, including various sizes, cloud-native hybrid variants, and cloud provider guidance, all underpinned by a continuous testing program.

6/20/2024

Introduced a new CI/CD Catalog component for Scheduled Reports Generation within Value Stream Management. This tool automates the creation and distribution of VSM reports by collecting metrics from projects/groups via the GraphQL API and presenting them in GitLab Flavored Markdown within issues. This enhances consistent monitoring, data-driven decision-making, and time savings for software managers.

1/18/2024

Introduced a DORA Performers score panel to the Value Streams Dashboard. This panel visualizes DevOps performance across projects by breaking down DORA performance levels into scores for each project under a group. This allows executives to identify performance highs and lows and understand organizational DevOps health.

6/26/2023

This post details the three-step analysis process for optimizing software value streams using GitLab's Value Stream Management (VSM) capabilities. It explains how the Value Streams Dashboard provides an organization-wide view of SDLC metrics, how project-specific Value Stream Analytics (VSA) allows for drilling down into individual project stages, and how to analyze specific work items within a stage to identify and fix bottlenecks. It emphasizes the actionable nature of these insights within the GitLab workflow.

6/12/2023

Introduces the GitLab Value Streams Dashboard, a customizable interface for decision-makers to identify trends, patterns, and opportunities for digital transformation improvements. It visualizes end-to-end DevSecOps workstreams and software development processes, providing a single source of truth for metrics. The first iteration focuses on benchmarking value stream lifecycle metrics, DORA metrics, and vulnerability metrics, with a comparison panel for month-to-month tracking and drill-down capabilities. The dashboard is customizable, allowing users to include panels from different subgroups or projects. The roadmap includes an executive-level summary, a DORA Performers score panel, and filtering by label.

6/8/2023

This post identifies and quantifies four best practices that contribute to faster software releases: cloud adoption, BizDevOps, CI/CD, and tracking DORA metrics. It cites survey data showing correlations between these practices and increased deployment frequency and release velocity. The post also touches on challenges organizations face in adopting these practices, such as complexity in cloud-native development and resistance to investing in CI/CD tools.

6/1/2023

Introduces the Value Stream Analytics Total Time Chart, a new visualization that breaks down the average time spent in each stage of the development workflow from idea to production. The chart uses a stacked area visualization to show stage contributions over time and includes tooltips for stage order. It is accessible via the Value Stream Analytics Overview page and integrates with other GitLab analytics pages and DORA metrics.

5/30/2023

Mughees Pervaiz updated expand/collapse buttons in roadmaps from link buttons to tertiary buttons, migrating them to new GitLab UI components and working with the tree view. Young Jun Joo implemented a memoization algorithm to cache search results, improving the search function's performance and efficiency by retrieving results from the cache on repeated searches.

4/13/2023

Introduced SourceWarp, a record-and-replay framework for source code management systems, to enable rapid, data-driven decision making for DevSecOps features. SourceWarp extracts relevant commits from Git history, generates a patch replay sequence, and replays it on target systems while monitoring for metrics. This was used to benchmark advanced vulnerability tracking, demonstrating a 30% improvement in effectiveness compared to traditional methods.

4/10/2023

Introduced rate limitations for unauthenticated users of the Projects List API to mitigate server load and ensure platform stability. The default limit is 400 requests per 10 minutes per IP address on GitLab.com, with configurable limits for self-managed instances.

2/14/2023

This post details specific technical optimizations applied to GitLab's 'View Source' page to reduce Total Blocking Time (TBT). It explains how TBT is measured and provides concrete examples of improvements made by leveraging CSS properties like `content-visibility: auto`, the Intersection Observer API, and VueJS directives like `v-once`. The post also discusses strategies for improving TBT within VueJS applications, such as optimizing props, using `v-memo`, and deferring non-critical tasks with `requestIdleCallback` or WebWorkers.

1/24/2023

This post introduces the beta release of the GitLab Value Streams Dashboard, which provides stakeholders with visibility into software development and delivery metrics (including DORA and flow metrics) to identify inefficiencies and bottlenecks. It also details the beta release of GitLab Remote Development, which enables developers to establish cloud-based development environments using a new Web IDE beta (based on VS Code) with an interactive terminal. The roadmap for Value Streams Dashboard includes new visualizations, security benchmarking, a data warehouse, and business value metrics. The roadmap for Remote Development includes provisioning environments on demand, sharing multi-repo environments, connecting from various IDEs, and integrating advanced security tools.

12/12/2022

This post details how automation within a DevSecOps platform benefits DevOps professionals by streamlining security (shifting left, automated testing), compliance (automated verification), configuration management, continuous integration (CI), continuous delivery (CD), and monitoring. It quantifies the increasing adoption of automation in testing and highlights how a unified DevSecOps platform reduces repetitive tasks, minimizes human error, and improves consistency, speed, and reliability in software development and deployment.

11/28/2022

This post details the diagnosis and resolution of chronic Redis latency spikes on GitLab.com. It describes the use of BPF and profiling tools (perf, flamescope, funclatency) to identify that eviction bursts were causing CPU saturation on the Redis main thread. The root cause was found to be an interaction effect leading to excessive evictions, and the solution involved a simple fix to break this cycle.

11/3/2022

This post introduces the concept of 'cadence' as a dominant factor in engineering organization performance, drawing parallels from aerospace (Gossamer Condor, SpaceX) and software development. It highlights the importance of high iteration rates and short cycle times, linking them to the DORA metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service) as indicators of engineering excellence. The post argues that an iterative approach, akin to the scientific method, allows for faster learning and improvement, leading to exponential gains over time.

8/30/2022

This post analyzes the 2022 Global DevSecOps Survey results, identifying code development, code review, security analysis, test data management, and testing as the top five reasons for software release delays. It highlights the increasing value developers place on code reviews and the growing adoption of AI in code review and testing. The post also discusses the challenges in keeping software secure and managing test data effectively, noting the increased investment in security and the growing adoption of SAST, DAST, and container scans.

7/19/2022

This post details the implementation of alert grouping by service and service dependency-based alert inhibition using Alertmanager to reduce pager noise for on-call engineers. It explains how alerts are grouped by the 'type' and 'env' labels, and how 'inhibition' rules are used to prevent cascading alerts when downstream services degrade. The post also outlines the process of defining service dependencies within the metrics-catalog and the restrictions applied to inhibit rules. The improvements observed include a reduction in pages per service and a downward trend in overall pages for the on-call.

6/20/2022

This post introduces the integration of DORA metrics (deployment frequency, lead time for changes, time to restore service, and change failure rate) into GitLab's Value Stream Analytics (VSA). It explains how VSA measures the median time spent by issues or merge requests in each development stage and how DORA metrics serve as key indicators of value stream output. The post details how VSA works by tracking workflow items through defined stages and provides examples of filtering and deep-diving into stage performance. It also highlights that full DORA metric visibility requires an Ultimate-tier account or a free trial.

5/9/2022

This post introduces an integration between GitLab CI and Nobl9 to enable SLO-as-Code. It details the steps to configure CI/CD variables (CLIENT_ID, CLIENT_SECRET, ACCESS_TOKEN, PROJECT, SLOCTL_YML) and provides a `.gitlab-ci.yml` configuration that uses the `nobl9/nobl9-ci-template` project to push SLO definitions to Nobl9 via the `sloctl` command-line tool. This automates the process of keeping SLO definitions synchronized between the code repository and Nobl9.

4/20/2022

This post introduces and explains the four key DORA metrics (Deployment Frequency, Lead Time for Changes, Time to Restore Service, Change Failure Rate) as a framework for measuring and improving DevOps team performance. It highlights how these metrics can drive visibility into the DevOps lifecycle, uncover bottlenecks, and influence decision-making for continuous improvement, citing Zoopla's use of GitLab for obtaining these metrics.

4/5/2022

This post introduces and details seven key DevOps metrics used at GitLab to measure engineering efficiency and productivity: master pipeline stability (target >95%), review app deployment success rate (target >99%), time to first failure (target <15 minutes), open S1 bug age (target <100 days), open S2 bug age (target <300 days), merge request pipeline duration (target <45 minutes), and MR pipeline costs (target <7.50). It explains the rationale behind each metric and its target, providing a concrete set of performance indicators for the engineering team.

2/7/2022

This post details the technical implementation of the pack-objects cache for Git fetch requests, which deduplicates identical requests to reduce CPU and memory usage. It also describes the development of an improved RPC transport for Git HTTP traffic, utilizing side channels to bypass gRPC overhead and reduce CPU usage on Gitaly servers. The post highlights the challenges faced, such as increased disk write throughput with the cache and potential gRPC memory allocation overhead, and the solutions implemented.

2/3/2022

This post details best practices for CI/CD, including automating everything, failing fast, committing frequently, shifting left on security, leveraging AI for pipeline troubleshooting, using feature flags, comprehensive monitoring, maintaining pipeline as code, and enabling feedback loops. It also covers continuous delivery best practices such as embracing deployment strategies (blue-green, canary, feature flags), maintaining environment parity, automating deployment validation, and practicing zero-downtime deployments. Optimization strategies for pipeline performance, visibility, resource efficiency, and scaling for team growth are discussed. Finally, it outlines how to measure CI/CD success using the DORA metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Mean Time to Recovery) and additional metrics like infrastructure costs, team retention, and business impact.

1/24/2022

This post introduces and details GitLab's Value Stream Analytics feature. It explains the concept of Value Stream Management (VSM) and its importance in optimizing software delivery by focusing on customer value and identifying bottlenecks. The post outlines the default DevOps stages tracked by Value Stream Analytics (Issue, Plan, Code, Test, Review, Staging), explains how to customize these stages at the group level, and highlights key metrics including DORA 4 metrics (Deployment Frequency, Lead Time for Changes). It emphasizes the benefit of GitLab's single-application approach for seamless data flow and analysis within the DevOps lifecycle.

1/24/2022

This post details how Zoopla adopted and implemented DORA metrics (Production deploy frequency, Lead time, Mean Time To Recover, Change fail rate, Time to onboard) to measure and improve their engineering performance as part of their 'Bedrock' replatforming program. They describe their data sources (GitLab, Blameless, Jenkins), the implementation effort involving Python and a standardized data schema, and the significant improvements achieved in deployment frequency, lead time, MTTR, change failure rate, and time to onboard. The post also highlights the cultural changes driven by these metrics and future plans for automation and wider adoption.

1/20/2022

This post details the implementation and impact of the Gitaly pack-objects cache, which significantly reduces CPU utilization during Git fetch operations. It contrasts this new solution with a previous custom CI pre-clone script, highlighting the advantages of the integrated cache. Performance benchmarks show a 3.6x improvement in benchmark run time and a reduction in average server CPU utilization from 100% to 40% when comparing GitLab 13.6 to GitLab 14.6 with the cache enabled.

12/15/2021

This post introduces and details the functionality of DevOps Reports and DevOps Score, new analytics features within GitLab. DevOps Reports provide insights into team adoption of key GitLab features across Development, Security, and Operations categories. DevOps Score allows for instance-level comparison of feature usage against leader benchmarks. Cohorts are also described as a tool for administrators to view user retention and manage seats. The post outlines the data sources (Service Ping) and availability tiers for these features.

10/26/2021

This post introduces and discusses the DORA metrics (Deployment Frequency, Lead Time for Changes, Time to Restore Service, and Change Failure Rate) as key indicators of DevOps team performance. It highlights the significant performance gap between elite and low-performing teams based on these metrics and provides six actionable tips for teams to improve their performance, including leveraging hybrid/multi-cloud environments, adhering to NIST cloud essentials, complementing DevOps with SRE practices, prioritizing documentation, integrating security throughout development, and fostering a positive team culture.

9/27/2021

This post details how to configure Sidekiq for specialized or large-scale GitLab instances. It covers identifying pain points through metrics (e.g., `sidekiq_queue_size`, `sidekiq_running_jobs`), monitoring Redis CPU usage, and customizing Sidekiq configurations. The recommended approach is to use routing rules and one-queue-per-shard for better clarity and reduced Redis load. It provides examples of using queue selectors to dedicate processing power to specific queues like `project_export` and discusses the trade-offs between queue selectors and routing rules.

9/2/2021

This post details the configuration of Sidekiq for large deployments on GitLab.com, focusing on addressing Redis CPU saturation. It explains the historical 'one queue per worker' approach and its limitations with a large number of queues. The post then describes the shift to a 'one queue per shard' model using Sidekiq routing rules, which significantly reduces Redis overhead by simplifying job fetching. It also touches upon Sidekiq's concurrency and threading model in Ruby.

8/5/2021

This post details the performance benchmarking of GitLab's 10,000 user reference architecture and GitLab Runner on AWS Graviton2 (Arm64) instances compared to x86-based instances. It quantifies cost savings (up to 23%) and performance gains (up to 36%) for the GitLab application and Runner, respectively. It describes the testing methodology using the GitLab Performance Tool and the GitLab Environment Toolkit, and presents CPU utilization graphs from Prometheus and Grafana to illustrate performance under load.

7/21/2021

This post introduces enhancements in GitLab 14 focused on providing better visibility into DevOps processes. It highlights the DORA metrics, specifically Deployment Frequency and Lead Time for Changes, and how GitLab 14 surfaces these metrics. It also details the integration with Value Stream Analytics, allowing users to identify and remediate blockers in their workflows directly from the platform.

4/28/2021

This post details the investigation and resolution of a production incident caused by Ruby 2.7's heap compaction feature when used with Puma's 'nakayoshi_fork'. It explains the concepts of tenuring objects and heap compaction, the challenges encountered with C-extensions like Hamlit due to improper object marking, and the specific fix implemented in Hamlit to resolve segfaults by statically resolving constants and using `rb_gc_register_mark_object`. It also highlights the need for more reliable detection of compaction-related issues in CI.

11/23/2020

This post details how GitLab's integrated monitoring capabilities, combined with advanced deployment strategies like canary deployments and feature flags, enable safe and low-risk releases. It highlights the Environment Dashboard for a cross-project view, the CD pipeline for incremental rollouts and performance testing, and the ability to perform rollbacks. It also explains the configuration and application of feature flags with different strategies and their combination with canary deployments, visualized via the deploy board. Finally, it showcases cluster-wide and pod-specific monitoring dashboards.

11/9/2020

This post details lessons learned by the Scalability Team in their project to investigate and improve Redis usage. Key takeaways include the importance of focusing on impact, applying Minimum Viable Change (MVC) principles to scaling problems, and structuring projects to deliver impact throughout. The team learned to prioritize tangible results over extensive visibility work and to iterate on smaller, impactful changes rather than a large, monolithic approach. They also emphasized updating documentation earlier in the process and proactively addressing issues with relevant teams.

10/27/2020

This post details GitLab's five-stage framework for optimizing infrastructure spend: 1. Basic cost visibility (using GCP billing exports), 2. Cost allocation (using GCP labels and handbook for infrastructure standards), 3. Optimize usage efficiency (identifying opportunities in CI runners, object storage lifecycle policies, and network usage), 4. Measure business outcomes vs spend (evaluating complex changes like advanced global search), and 5. Predict future spend and problem areas (still a work in progress). It also outlines the workflow and planning process for cost optimization initiatives, prioritizing based on cost savings, customer impact, future potential cost impact, and effort required.

8/27/2020

This post introduces and details the Merge Request (MR) Rate as a key performance indicator for measuring engineering productivity at GitLab. It defines MR Rate, discusses its rationale (incentivizing iteration, faster feature delivery, codebase improvement), outlines challenges (quantity over quality, balancing team vs. individual metrics), and explains strategies for increasing it (improving iteration, setting KPIs, setting OKRs, empowering teams). It also mentions other related metrics like Product MRs by Type and Say Do Ratio.

8/18/2020

This post advocates for the 'boring solutions' philosophy in software development, emphasizing the choice of established, popular technologies over novel or complex ones to reduce overall system complexity and enable faster iteration. It provides several examples from GitLab's history, including using issue labels for issue boards, using `curl` against API endpoints instead of building new UIs, using JSON Web Tokens for authentication, and using SQL for the CI job queue. It also mentions the migration from Gitolite to GitLab Shell as an example of abandoning a 'boring' solution when it no longer worked.

5/6/2020

This post introduces the concept of iterative development, inspired by John Boyd's iteration law, as a means to increase the merge request rate. It emphasizes a bias for action, accepting that 'all code is bad code' and embracing course correction, and the importance of writing things down and tightening feedback loops to foster a team environment that allows for rapid iteration and continuous improvement.

2/18/2020

The Quality team developed and released the GitLab Performance tool (GPT), built on k6, to automate load testing of GitLab environments. GPT includes a broad test suite, customization options, sequential test execution, enhanced reporting, and built-in success thresholds. It is integrated into GitLab CI pipelines for daily/weekly testing against reference architectures and for release comparison tests. The tool has helped identify performance bottlenecks and drive improvements, with results published on the GPT wiki.

1/30/2020

This post details a method to significantly reduce the file size of screenshots through color quantization (using pngquant) and DEFLATE compression (using zopfli). It provides command-line instructions and automation scripts for macOS using Hazel and Automator, demonstrating how to achieve up to 80% file size reduction. The technical details include explanations of PNG-32 vs. PNG-8, color quantization, and the benefits of zopfli over standard zlib.

1/16/2020

This post announces the migration of GitLab.com's CDN from Fastly to Cloudflare. The primary technical drivers for this migration are to improve availability, security, and performance by leveraging Cloudflare's Web Application Firewall (WAF), Spectrum (for port 22 traffic), and Argo Smart Routing. The migration is complex because GitLab.com serves both HTTPS and SSH (port 22) traffic, requiring a CDN solution capable of handling both. The post details the timing of the switch (March 28, 2020), the need for users with firewalled access to update their IP whitelists to Cloudflare's ranges, and potential impacts on custom/private runners due to DNS or SSL certificate caching. It also mentions the use of a readiness review to plan the transition.

12/16/2019

This post introduces a framework for optimizing the value exchange system by shifting focus from output to outcomes, analyzing value streams, and applying continuous improvement principles (Kaizen) to identify and address bottlenecks in the R&D process. It uses a case study of Acme Co. to illustrate how a decline in revenue growth was traced back to slow capability delivery from R&D, which was a result of misaligned metrics (focusing on output over outcomes) and a lack of understanding of the value delivery stream. The post advocates for measuring value delivery streams and applying the Improvement Kata to achieve target conditions.

12/16/2019

This post details the identification and reduction of waste in the R&D value stream, specifically focusing on reducing 'waiting' time through process improvements like pair programming and work-in-progress limits, leading to a significant reduction in lead time. It also highlights the critical impact of technical debt and defects on the delivery of new value, leading to a strategic shift to address these issues.

12/16/2019

This post introduces a new approach to optimizing the value exchange system by adopting throughput accounting and an intelligent decision support framework using NLP to extract topics and sentiment from text-based data, enabling better prioritization of the roadmap and a shift from measuring output to measuring outcomes through continuous innovation and lean sprints.

11/26/2019

This post details the author's personal journey and strategies for improving productivity by increasing 'deep work' and 'E-factor' (uninterrupted hours / body-present hours). It introduces concepts from 'Deep Work' and 'Peopleware', and describes personal workflow adjustments (email timing, self-improvement activities, routine) and environmental factors (separate room, routine, timezone, Slack/email policy, task journaling, asynchronous work) that enabled these improvements. The author shares a Google Spreadsheet template for time tracking and a GitLab project for task journaling as tools to support these practices.

11/4/2019

This post introduces a framework for assessing data organization maturity, categorizing data analysis into three tiers: reporting, insights, and predictions. It emphasizes that a mature data organization is foundational to a mature analytics organization. The post details the evolution of data analysis from gatekept access to democratized data, highlighting the value of insights derived from combining data sources. It also provides guidance on maturing data teams through investment, empowerment, and leveraging open-source analytics tools like dbt for speed to value and best practices.

10/14/2019

Introduced an asynchronous approach to update namespace storage statistics using Sidekiq jobs and Redis for lease management. This avoids increasing the transaction length of project statistics updates and improves performance for large namespaces. The solution involves creating a `namespace_aggregation_schedules` table, enqueuing Sidekiq jobs for immediate and delayed execution, and using Redis leases to prevent concurrent aggregations. A cron job is also scheduled to process any remaining pending updates.

7/24/2019

The Support team has developed and open-sourced three tools: strace-parser, JSON Stats, and GitLabSOS. Strace-parser summarizes strace output to identify performance bottlenecks by analyzing syscalls, file operations, and their durations. JSON Stats aggregates performance metrics from GitLab's internal stats endpoint. GitLabSOS is a diagnostic tool that collects relevant system and GitLab configuration information for troubleshooting. These tools aim to reduce the friction in analyzing performance-related issues for self-managed customers by providing lightweight, easily deployable solutions.

4/29/2019

This post introduces strategies for managing technical debt, emphasizing documentation, embracing Minimum Viable Change (MVC), and leveraging CI/CD automation for continuous delivery to pay down debt incrementally.

3/20/2019

This post details efforts to improve Elasticsearch integration for global code search on GitLab.com. Key technical challenges and attempted solutions include reducing Elasticsearch index size by investigating force merges (abandoned due to performance degradation), adjusting Edge NGram sizes for blobs (potential 7-15% savings, pending validation), and exploring separate indexes (no space savings but improved indexing speed). Administration capabilities are being enhanced with better cluster visibility, graceful recovery from indexing failures, selective/progressive indexing, making code indexing optional, and sharding Elasticsearch per group.

3/4/2019

This post introduces a new perspective on 'cycle time' in software development, advocating for its measurement to include the planning stage alongside the traditional engineering-focused definition (from starting work on an issue to production deployment). It emphasizes that reducing this end-to-end cycle time is a competitive advantage, enabling faster delivery of value to customers and better market responsiveness. The post suggests increasing visibility and adopting CI/CD as key strategies for cycle time reduction.

10/12/2018

This post details ten strategies for reducing development cycle times, drawing from the experiences of GitLab engineering leads. Key strategies include automating work (CI/CD, manual tasks), breaking work into smaller iterative pieces, utilizing feature flags, measuring the impact of workflow components, improving cross-team communication and automation, and managing asynchronous work by having multiple items to work on. It also touches upon challenges like getting it right the first time, differentiating helpful from unhelpful processes, working across teams, and asynchronous communication.

3/14/2018

The production team developed an 'on-call robot assistant' program that automates the generation of on-call handover reports. This program pulls data from PagerDuty (incidents), GitLab issues (production backlog), and GitLab Prometheus (key performance metrics) to create a GitLab issue for the handover meeting. This automates tedious work and ensures consistency in reviewing incidents and metrics.

1/23/2018

This post details the debugging and resolution of performance degradation in the Gitaly service. The issue was identified as increasing latency over time, eventually leading to lockups. Through profiling (pprof) and cAdvisor monitoring, contention on `syscall.ForkLock` was pinpointed. Research led to the discovery that Go 1.9's switch from `fork/exec` to `posix_spawn` resolved this by reducing the overhead of process creation, especially with large virtual memory footprints. Recompiling Gitaly with Go 1.9 resulted in a 30x reduction in latency.

9/26/2016

This post details the challenges and solutions related to scaling GitLab.com's infrastructure, specifically addressing storage issues with CephFS and PostgreSQL vacuuming performance under high load. The company is committed to documenting these solutions to help other organizations facing similar scaling challenges.

9/21/2016

Introduced Cycle Analytics to measure and visualize the time it takes for a team to go from an idea to production. This feature breaks down the development process into distinct stages: Issue, Plan, Code, Test, Review, Staging, and Production. It relies on GitLab CI configuration and specific commit message patterns (e.g., 'Closes #xxx') to track progress through these stages. The initial iteration focuses on measuring median times for each stage and the total cycle time.

8/12/2016

This post details a philosophical shift from an 'Operations mindset' to an 'Infrastructure mindset' for managing GitLab's performance and scalability. Key steps include focusing on infrastructure from a development perspective, spotting patterns through time-based metrics, investigating performance gaps, aligning resources effectively, automating manual tasks, and iterating on solutions. It provides a case study on improving SSH Git access time by identifying and addressing bottlenecks in SSH authorization, worker restarts, HTTP request queuing, and DBPUS connection limits. The post also lists the toolbox used for these efforts, including Chef, Prometheus, Blackbox exporter, InfluxDB, Grafana, ELK stack, and Sentry.

2/25/2016

This post details specific performance improvements implemented in GitLab 8.5, focusing on optimizing the fetching of issues closed by a merge request and improving the performance of retrieving last update times for events. It introduces the use of InfluxDB and Grafana for production performance monitoring and Sherlock for development-only monitoring, along with the benchmark-ips library. The post elaborates on the technical details of the `MergeRequest#closes_issues` optimization, which involved concatenating commit messages to reduce database queries, and the optimization of event data retrieval for Atom feeds by refining SQL queries. It also touches upon optimizing file handling by conditionally setting autocrlf.

6/5/2015

This post details GitLab's use of Unicorn as a pre-forking Ruby web server for handling web requests and Git HTTP clients. It explains Unicorn's multi-process architecture for CPU utilization and fault tolerance, including how the master process spawns and replaces worker processes that crash or time out. It also introduces the unicorn-worker-killer gem, a solution implemented to manage memory leaks in Unicorn workers by self-checking memory usage after a set number of requests and exiting if a threshold is exceeded, allowing the Unicorn master to replace the worker without dropping user requests. The post provides log examples of both worker timeouts and memory limit exits.