TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

Performance Optimization

70 posts · Jun 2015 – Sep 2026

What this trail covers

GitLab CI/CD pipelines offer fine-grained permissions for job tokens, moving from Beta to General Availability. This enhancement allows maintainers to implement granular access controls for job tokens to API resources, adhering to the principle of least privilege. The initial release supports permissions for repositories, deployments, environments, jobs, packages, pipelines, releases, secure files, and Terraform state. This feature aims to reduce the attack surface, eliminate reliance on lon. This post details implementing continuous deployment with GitLab, focusing on artifact management strategies (versioning, retention, registry access), environment strategy, and deployment targets. It provides a step-by-step example for containerizing a simple HTML application, setting up a CI/CD pipeline with Docker, publishing to the GitLab Container Registry, and deploying to a virtual machine using SSH authentication. The pipeline includes stages for publishing, staging, and production deployments, with environment configuration for tracking.

All GitLab trails
  1. How to design GitLab for enterprise scaleThis post details architectural decisions for scaling GitLab at the enterprise level, focusing on deployment models (GitLab.com, GitLab Dedicated, Self-Managed), runner strategy planning (sizing, scope, autoscaling, executors), and high availability/disaster recovery planning (SLOs, RTO, RPO, GitLab Geo). It also outlines where pipeline performance breaks down at scale, emphasizing the importance of establishing performance baselines and testing large repositories/monorepos separately. Key technical considerations include capacity planning for runner fleets based on workload rather than headcount, matching executors (Kubernetes, Docker Autoscaler, Instance) to workload requirements, and defining redundancy and recovery strategies for self-managed deployments. The post also mentions Kubernetes and cloud-native deployment considerations.
  2. Rate limits on GitLab.com are changingThis post details the upcoming changes to GitLab.com's rate limits, aligning them with subscription tiers (Free, Premium, Ultimate) and differentiating between authenticated and unauthenticated requests. It outlines the phased rollout schedule, provides guidance on how users can stay within limits (authentication, batching, caching, pagination, exponential backoff), and clarifies what does and does not change. Preview windows (brownouts) are announced for testing the new limits. The post also mentions future plans for purchasing additional capacity above standard limits.
  3. How we used AI agents to migrate GitLab rate limitingThis post details the migration of GitLab's legacy rate-limiting system to a unified implementation in `labkit-ruby` using AI agents. It describes the setup involving a pod of engineers and AI agents, the strict loop of spec, adversarial review, implementation, and gradual rollout. Key successes include the mechanical fan-out of changes across numerous call sites by agents and the identification of bottlenecks in review capacity and operator attention. Challenges encountered include a shadow-mode miss where an adapter dropped an identifier, a lack of distinction between structural collisions and normal disagreements in observability, and Redis cluster capacity limitations. The post highlights the importance of human judgment in defining "done" and the need for comprehensive audits to ensure all rate limiting keys are migrated. The migration of the `RackAttack` layer is the next step.
  4. Optimize GitLab object storage for scale and performanceThis post details how to optimize GitLab's object storage configuration for scale and performance. It introduces a consolidated form for configuring artifacts, LFS, uploads, and packages, emphasizing the use of IAM roles for authentication and server-side encryption. It also covers separate configuration for the container registry using the s3_v2 driver and disabling proxy downloads for direct client downloads from object storage. Key differences between consolidated and separate configurations are highlighted, along with a migration path.
  5. Fine-grained permissions for job tokens is now GAThis post announces the General Availability (GA) of fine-grained permissions for CI/CD job tokens. Previously, job tokens inherited overprivileged permissions from user accounts. This GA release introduces explicit control over job token access to specific API resources, including repositories, deployments, environments, jobs, packages, pipelines, releases, secure files, and Terraform state. This is an opt-in capability designed to enhance software supply chain security by implementing the principle of least privilege and reducing the need for long-lived tokens.
  7. Exact Code Search: Find code faster across repositoriesIntroduced Exact Code Search, a new capability powered by Zoekt, to provide lightning-fast and precise code search across large codebases. This feature leverages positional trigrams for exact matches, optimized regular expression performance, and a distributed, self-registering node architecture with intelligent load balancing and seamless security integration. It addresses challenges of traditional code search by eliminating false positives and providing contextual multi-line results, significantly improving debugging, code exploration, refactoring, and security auditing workflows.
  8. Improving OAuth ROPC security on GitLab.comThis post announces a mandatory security enhancement for OAuth Resource Owner Password Credentials (ROPC) on GitLab.com, requiring client authentication for all requests starting April 8, 2025. It details the reasons for the change (enhanced security, standards compliance, improved auditing) and provides required actions for users to update their integrations by registering applications to obtain client credentials (`client_id`, `client_secret`) and updating their token requests. Examples of insecure and required ROPC request methods are provided.
  9. Prepare now: Docker Hub rate limits will impact GitLab CI/CDThis post details upcoming Docker Hub rate limits that will impact CI/CD pipelines, particularly for unauthenticated pulls. GitLab is responding by enhancing the Dependency Proxy with authentication support and updating documentation. Users are advised to configure Docker Hub authentication in their pipelines, use the GitLab Container Registry, or leverage the Dependency Proxy with authentication to mitigate these changes. Best practices like using specific image tags and consolidating Dockerfiles are also recommended.
  10. From code to production: A guide to continuous deployment with GitLabThis post details implementing continuous deployment with GitLab, focusing on artifact management strategies (versioning, retention, registry access), environment strategy, and deployment targets. It provides a step-by-step example for containerizing a simple HTML application, setting up a CI/CD pipeline with Docker, publishing to the GitLab Container Registry, and deploying to a virtual machine using SSH authentication. The pipeline includes stages for publishing, staging, and production deployments, with environment configuration for tracking.
  11. Data-driven DevSecOps: Exploring GitLab Insights DashboardsThis post details how to leverage GitLab Insights Dashboards for data-driven DevSecOps. It explains the various analytics tools available within GitLab (Productivity Analytics, Code Review Analytics, CI/CD Analytics, Value Stream Analytics, Insights) and emphasizes the role of Labels in filtering and focusing metrics. It provides a step-by-step guide on configuring Insights Dashboards using `.gitlab/insights.yml` files for both projects and groups, and offers detailed examples of customizing merge request insights to track squad productivity and requirement types. The post also includes hints on automating label application and provides sample YAML configurations for different dashboard views.
  12. How we designed the GitLab Reference ArchitecturesThis post details the design and evolution of GitLab Reference Architectures, which provide scalable and elastic starting points for deploying GitLab at scale. It outlines the problem of customer hurdles in deploying GitLab at scale, the goals of performance, availability, scalability, elasticity, cost-effectiveness, and maintainability, and the iterative process of design, validation, and refinement. Key aspects include collecting metrics like Requests per Seconds (RPS) from existing environments, defining prototype architectures based on GitLab.com's design, and using tools like the GitLab Performance Tool and GitLab Environment Toolkit for automated environment building and testing. The post highlights the continuous improvement of these architectures with new sizes, cloud-native variants, and cloud provider guidance, all supported by a comprehensive weekly testing program.
  13. How visualization improves the GitLab merge train experienceIntroduced merge train visualization to provide users with a clear view of the merge queue, including lists of queued and merged MRs, and the ability to remove MRs directly from the visualization. This aims to enhance transparency, trust, efficiency, collaboration, and reduce the risk of failures.
  14. FAQ: The RegreSSHion vulnerability and GitLabThis post details the RegreSSHion vulnerability (CVE-2024-6387) in OpenSSH and its potential impact on self-managed GitLab instances. It clarifies that GitLab.com and GitLab Dedicated are not affected due to the use of gitlab-sshd. For self-managed instances, it recommends applying OS patches, using gitlab-sshd, and following GitLab Hardening Recommendations. It also discusses the low real-world exploitability of the vulnerability due to its nature and the difficulty of successful execution.
  15. New Scheduled Reports Generation tool simplifies value stream managementIntroduced a new CI/CD Catalog component for Scheduled Reports Generation within Value Stream Management. This tool automates the creation and distribution of VSM reports by collecting metrics from projects/groups via the GraphQL API and presenting them in GitLab Flavored Markdown within issues. This enhances consistent monitoring, data-driven decision-making, and time savings for software managers.
  16. Rate limitations announced for Projects, Groups, and Users APIsIntroduces rate limitations for the Projects, Groups, and Users APIs to manage server load and platform stability. Details a phased rollout strategy including brownouts and permanent enforcement dates. Specifies how rate limits will be managed for Self-managed and Dedicated instances via feature flags, with default limits applied to new installations from GitLab 18.3. Lists specific rate limits per endpoint for authenticated users (per user) and unauthenticated requests (per IP). Explains the '429 Too Many Requests' response and how Self-managed admins can configure limits.
  17. Inside DORA Performers score in GitLab Value Streams DashboardIntroduced a DORA Performers score panel to the Value Streams Dashboard. This panel visualizes DevOps performance across projects by breaking down DORA performance levels into scores for each project under a group. This allows executives to identify performance highs and lows and understand organizational DevOps health.
  18. Unmasking password attacks at GitLabThis post details an increase in password attacks against the GitLab.com OAuth API endpoint, starting September 22, 2023. It describes the automated nature of these attacks and the target of simple usernames. GitLab's security controls are effectively mitigating these attacks. The post recommends enabling two-factor authentication and highlights a product update to reduce account lockout risks for users with 2FA. It also advises enforcing GitLab-layer 2FA, using IP address restrictions for group access, and implementing Git Abuse Rate Limiting. General security advice includes using strong, unique passwords and vigilance against phishing.
  19. GitLab's 3 steps to optimizing software value streamsThis post details the three-step analysis process for optimizing software value streams using GitLab's Value Stream Management (VSM) capabilities. It explains how the Value Streams Dashboard provides an organization-wide view of SDLC metrics, how project-specific Value Stream Analytics (VSA) allows for drilling down into individual project stages, and how to analyze specific work items within a stage to identify and fix bottlenecks. It emphasizes the actionable nature of these insights within the GitLab workflow.
  20. Getting started with the new GitLab Value Streams DashboardIntroduces the GitLab Value Streams Dashboard, a customizable interface for decision-makers to identify trends, patterns, and opportunities for digital transformation improvements. It visualizes end-to-end DevSecOps workstreams and software development processes, providing a single source of truth for metrics. The first iteration focuses on benchmarking value stream lifecycle metrics, DORA metrics, and vulnerability metrics, with a comparison panel for month-to-month tracking and drill-down capabilities. The dashboard is customizable, allowing users to include panels from different subgroups or projects. The roadmap includes an executive-level summary, a DORA Performers score panel, and filtering by label.
  21. Value stream optimization with GitLab's Total Time ChartIntroduces the Value Stream Analytics Total Time Chart, a new visualization that breaks down the average time spent in each stage of the development workflow from idea to production. The chart uses a stacked area visualization to show stage contributions over time and includes tooltips for stage order. It is accessible via the Value Stream Analytics Overview page and integrates with other GitLab analytics pages and DORA metrics.
  22. SourceWarp: Make data-driven, agile DevSecOps decisionsIntroduced SourceWarp, a record-and-replay framework for source code management systems, to enable rapid, data-driven decision making for DevSecOps features. SourceWarp extracts relevant commits from Git history, generates a patch replay sequence, and replays it on target systems while monitoring for metrics. This was used to benchmark advanced vulnerability tracking, demonstrating a 30% improvement in effectiveness compared to traditional methods.
  23. Rate limitations for unauthorized users of the Projects List APIIntroduces rate limiting for unauthenticated users of the Projects List API to mitigate server load and ensure platform stability. The limit is set to 400 requests per 10 minutes per unique IP address on GitLab.com, with self-managed instances allowing admin customization. Authorized users are unaffected.
  24. Total Blocking Time - The metric to know for faster website performanceThis post details the technical approach to reducing Total Blocking Time (TBT) on GitLab's 'View Source' page. It explains TBT and long tasks, demonstrates measurement using Chrome DevTools, and provides specific optimization techniques. These include changing UX (Show More, paging, virtual scrolling), lazy-loading images and lists, reducing excessive HTML, deferring rendering with `content-visibility: auto` and Intersection Observer API, and using `requestIdleCallback`. For VueJS, it suggests Server-side rendering (SSR) or streaming, avoiding unnecessary component abstractions, optimizing props, using `v-memo` and `v-once`, and deferring non-critical tasks with `requestIdleCallback` or WebWorkers. The post quantifies the improvement from ~3 seconds to ~500 ms TBT using `content-visibility: auto`, Intersection Observer API, and `v-once`.
  25. The GitLab Quarterly: How our latest beta releases support developersThis post introduces the beta release of the GitLab Value Streams Dashboard, which provides stakeholders with visibility into software development and delivery metrics (including DORA and flow metrics) to identify inefficiencies and bottlenecks. It also details the beta release of GitLab Remote Development, which enables developers to establish cloud-based development environments using a new Web IDE beta (based on VS Code) with an interactive terminal. The roadmap for Value Streams Dashboard includes new visualizations, security benchmarking, a data warehouse, and business value metrics. The roadmap for Remote Development includes provisioning environments on demand, sharing multi-repo environments, connecting from various IDEs, and integrating advanced security tools.
  26. How we diagnosed and resolved Redis latency spikes with BPF and other toolsThis post details the diagnosis and resolution of chronic Redis latency spikes on GitLab.com. It describes the use of BPF and profiling tools (perf, flamescope, funclatency) to identify that eviction bursts were causing CPU saturation on the Redis main thread. The root cause was found to be an interaction effect leading to excessive evictions, and the solution involved a simple fix to break this cycle.
  27. Cadence is everything: 10x engineering organizations for 10x engineersThis post introduces the concept of 'cadence' as a dominant factor in engineering organization performance, drawing parallels from aerospace (Gossamer Condor, SpaceX) and software development. It highlights the importance of high iteration rates and short cycle times, linking them to the DORA metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Time to Restore Service) as indicators of engineering excellence. The post argues that an iterative approach, akin to the scientific method, allows for faster learning and improvement, leading to exponential gains over time.
  28. How we improved on-call life by reducing pager noiseThis post details the implementation of alert grouping by service and service dependency-based alert inhibition using Alertmanager to reduce pager noise for on-call engineers. It explains how alerts are grouped by the 'type' and 'env' labels, and how 'inhibition' rules are used to prevent cascading alerts when downstream services degrade. The post also outlines the process of defining service dependencies within the metrics-catalog and the restrictions applied to inhibit rules. The improvements observed include a reduction in pages per service and a downward trend in overall pages for the on-call.
  29. Improving visibility: GitLab's value stream and DORA metricsThis post introduces the integration of DORA metrics (deployment frequency, lead time for changes, time to restore service, and change failure rate) into GitLab's Value Stream Analytics (VSA). It explains how VSA measures the median time spent by issues or merge requests in each development stage and how DORA metrics serve as key indicators of value stream output. The post details how VSA works by tracking workflow items through defined stages and provides examples of filtering and deep-diving into stage performance. It also highlights that full DORA metric visibility requires an Ultimate-tier account or a free trial.
  30. Enable SLO-as-Code with Nobl9 and GitLabThis post introduces an integration between GitLab CI and Nobl9 to enable SLO-as-Code. It details the steps to configure CI/CD variables (CLIENT_ID, CLIENT_SECRET, ACCESS_TOKEN, PROJECT, SLOCTL_YML) and provides a `.gitlab-ci.yml` configuration that uses the `nobl9/nobl9-ci-template` project to push SLO definitions to Nobl9 via the `sloctl` command-line tool. This automates the process of keeping SLO definitions synchronized between the code repository and Nobl9.
  31. How the DORA metrics can help DevOps team performanceThis post introduces and explains the four key DORA metrics (Deployment Frequency, Lead Time for Changes, Time to Restore Service, Change Failure Rate) as a framework for measuring and improving DevOps team performance. It highlights how these metrics can drive visibility into the DevOps lifecycle, uncover bottlenecks, and influence decision-making for continuous improvement, citing Zoopla's use of GitLab for obtaining these metrics.
  32. The top DevOps tooling metrics and targets at GitLabThis post introduces and details seven key DevOps metrics used at GitLab to measure engineering efficiency and productivity: master pipeline stability (target >95%), review app deployment success rate (target >99%), time to first failure (target <15 minutes), open S1 bug age (target <100 days), open S2 bug age (target <300 days), merge request pipeline duration (target <45 minutes), and MR pipeline costs (target <7.50). It explains the rationale behind each metric and its target, providing a concrete set of performance indicators for the engineering team.
  33. Git fetch performance improvements in 2021, Part 2This post details the technical implementation of the pack-objects cache for Git fetch requests, which deduplicates identical requests to reduce CPU and memory usage. It also describes the development of an improved RPC transport for Git HTTP traffic, utilizing side channels to bypass gRPC overhead and reduce CPU usage on Gitaly servers. The post highlights the challenges faced, such as increased disk write throughput with the cache and potential gRPC memory allocation overhead, and the solutions implemented.
  34. How to keep up with CI/CD best practicesThis post details best practices for CI/CD, including automating everything, failing fast, committing frequently, shifting left on security, leveraging AI for pipeline troubleshooting, using feature flags, comprehensive monitoring, maintaining pipeline as code, and enabling feedback loops. It also covers continuous delivery best practices such as embracing deployment strategies (blue-green, canary, feature flags), maintaining environment parity, automating deployment validation, and practicing zero-downtime deployments. Optimization strategies for pipeline performance, visibility, resource efficiency, and scaling for team growth are discussed. Finally, it outlines how to measure CI/CD success using the DORA metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Mean Time to Recovery) and additional metrics like infrastructure costs, team retention, and business impact.
  35. The role of Value Stream Analytics in GitLab's DevOps PlatformThis post introduces and details GitLab's Value Stream Analytics feature. It explains the concept of Value Stream Management (VSM) and its importance in optimizing software delivery by focusing on customer value and identifying bottlenecks. The post outlines the default DevOps stages tracked by Value Stream Analytics (Issue, Plan, Code, Test, Review, Staging), explains how to customize these stages at the group level, and highlights key metrics including DORA 4 metrics (Deployment Frequency, Lead Time for Changes). It emphasizes the benefit of GitLab's single-application approach for seamless data flow and analysis within the DevOps lifecycle.
  36. Zoopla Boosts Deployments & Automation with DORA MetricsThis post details how Zoopla adopted and implemented DORA metrics (Production deploy frequency, Lead time, Mean Time To Recover, Change fail rate, Time to onboard) to measure and improve their engineering performance as part of their 'Bedrock' replatforming program. They describe their data sources (GitLab, Blameless, Jenkins), the implementation effort involving Python and a standardized data schema, and the significant improvements achieved in deployment frequency, lead time, MTTR, change failure rate, and time to onboard. The post also highlights the cultural changes driven by these metrics and future plans for automation and wider adoption.
  37. How we made Git fetch performance improvements in 2021, part 1This post details the implementation and impact of the Gitaly pack-objects cache, which significantly reduces CPU utilization during Git fetch operations. It contrasts this new solution with a previous custom CI pre-clone script, highlighting the advantages of the integrated cache. Performance benchmarks show a 3.6x improvement in benchmark run time and a reduction in average server CPU utilization from 100% to 40% when comparing GitLab 13.6 to GitLab 14.6 with the cache enabled.
  38. Understand how your teams adopt DevOps with DevOps reportsThis post introduces and details the functionality of DevOps Reports and DevOps Score, new analytics features within GitLab. DevOps Reports provide insights into team adoption of key GitLab features across Development, Security, and Operations categories. DevOps Score allows for instance-level comparison of feature usage against leader benchmarks. Cohorts are also described as a tool for administrators to view user retention and manage seats. The post outlines the data sources (Service Ping) and availability tiers for these features.
  39. How to make your DevOps team elite performersThis post introduces and discusses the DORA metrics (Deployment Frequency, Lead Time for Changes, Time to Restore Service, and Change Failure Rate) as key indicators of DevOps team performance. It highlights the significant performance gap between elite and low-performing teams based on these metrics and provides six actionable tips for teams to improve their performance, including leveraging hybrid/multi-cloud environments, adhering to NIST cloud essentials, complementing DevOps with SRE practices, prioritizing documentation, integrating security throughout development, and fostering a positive team culture.
  40. How to configure Sidekiq for specialized or large-scale GitLab instancesThis post details how to configure Sidekiq for specialized or large-scale GitLab instances. It covers identifying pain points through metrics (e.g., `sidekiq_queue_size`, `sidekiq_running_jobs`), monitoring Redis CPU usage, and customizing Sidekiq configurations. The recommended approach is to use routing rules and one-queue-per-shard for better clarity and reduced Redis load. It provides examples of using queue selectors to dedicate processing power to specific queues like `project_export` and discusses the trade-offs between queue selectors and routing rules.
  41. What we learned about configuring Sidekiq from GitLab.comThis post details the configuration of Sidekiq for large deployments on GitLab.com, focusing on addressing Redis CPU saturation. It explains the historical 'one queue per worker' approach and its limitations with a large number of queues. The post then describes the shift to a 'one queue per shard' model using Sidekiq routing rules, which significantly reduces Redis overhead by simplifying job fetching. It also touches upon Sidekiq's concurrency and threading model in Ruby.
  42. GitLab on Graviton2: 23% cheaper, 36% higher performanceThis post details the performance benchmarking of GitLab's 10,000 user reference architecture and GitLab Runner on AWS Graviton2 (Arm64) instances compared to x86-based instances. It quantifies cost savings (up to 23%) and performance gains (up to 36%) for the GitLab application and Runner, respectively. It describes the testing methodology using the GitLab Performance Tool and the GitLab Environment Toolkit, and presents CPU utilization graphs from Prometheus and Grafana to illustrate performance under load.
  43. Optimize DevOps with enhanced visibility tools in GitLab 14This post introduces enhancements in GitLab 14 focused on providing better visibility into DevOps processes. It highlights the DORA metrics, specifically Deployment Frequency and Lead Time for Changes, and how GitLab 14 surfaces these metrics. It also details the integration with Value Stream Analytics, allowing users to identify and remediate blockers in their workflows directly from the platform.
  44. Ruby 2.7: Understand and debug problems with heap compactionThis post details the investigation and resolution of a production incident caused by Ruby 2.7's heap compaction feature when used with Puma's 'nakayoshi_fork'. It explains the concepts of tenuring objects and heap compaction, the challenges encountered with C-extensions like Hamlit due to improper object marking, and the specific fix implemented in Hamlit to resolve segfaults by statically resolving constants and using `rb_gc_register_mark_object`. It also highlights the need for more reliable detection of compaction-related issues in CI.
  45. Automation check-in and rate limit changes on GitLab.comIntroduced new rate limits for GitLab.com: 500 requests/minute for unauthenticated IP traffic, 2000 requests/minute for authenticated API traffic per user, 1000 requests/minute for other authenticated HTTP traffic per user, and 2000 requests/minute for all other IP traffic. Documented handling of 429 status codes and Retry-After headers for automated traffic.
  46. GitLab's unifiied and integrated monitoring strategiesThis post details how GitLab's integrated monitoring capabilities, combined with advanced deployment strategies like canary deployments and feature flags, enable safe and low-risk releases. It highlights the Environment Dashboard for a cross-project view, the CD pipeline for incremental rollouts and performance testing, and the ability to perform rollbacks. It also explains the configuration and application of feature flags with different strategies and their combination with canary deployments, visualized via the deploy board. Finally, it showcases cluster-wide and pod-specific monitoring dashboards.
  47. How to make Docker Hub rate limit monitoring a breezeThis post introduces methods for monitoring Docker Hub rate limits, which can impact CI/CD pipelines and production deployments. It details how to check current limits using `curl` and `jq`, and provides a Python script (`check_docker_hub_limit.py`) for integration with monitoring systems like Nagios/Icinga/Sensu/Zabbix. Additionally, it describes a Prometheus exporter (`docker-hub-limit-exporter`) for exposing Docker Hub rate limit metrics, enabling visualization and trend analysis. The post also references existing GitLab blog posts on caching Docker images and using the Dependency Proxy as complementary solutions.
  48. Lessons in iteration from a new team in infrastructureThis post details lessons learned by the Scalability Team in their project to investigate and improve Redis usage. Key takeaways include the importance of focusing on impact, applying Minimum Viable Change (MVC) principles to scaling problems, and structuring projects to deliver impact throughout. The team learned to prioritize tangible results over extensive visibility work and to iterate on smaller, impactful changes rather than a large, monolithic approach. They also emphasized updating documentation earlier in the process and proactively addressing issues with relevant teams.
  49. How we optimized infrastructure spend at GitLabThis post introduces a five-stage framework for infrastructure spend optimization at GitLab, detailing each stage with specific examples. Stage 1 (Basic Cost Visibility) involves using GCP's Default Billing Export to understand high-level spend. Stage 2 (Cost Allocation) uses GCP labels and handbook standards to tie costs to business metrics like CI runners and employee usage. Stage 3 (Optimize Usage Efficiency) highlights specific optimizations for CI runners, object storage (reducing 900TB of Postgres backups), and network usage (downloading runner artifacts directly from GCS). Stage 4 (Measure Business Outcomes vs Spend) discusses analyzing complex changes like the rollout of advanced global search. Stage 5 (Predict Future Spend) is an ongoing effort. The post also outlines the workflow and planning process using an issue board and prioritization factors (cost savings, customer impact, future potential cost impact, effort required).
  50. How we measure engineering productivity at GitLabThis post introduces and details the Merge Request (MR) Rate as a key performance indicator for measuring engineering productivity at GitLab. It defines MR Rate, discusses its rationale (incentivizing iteration, faster feature delivery, codebase improvement), outlines challenges (quantity over quality, balancing team vs. individual metrics), and explains strategies for increasing it (improving iteration, setting KPIs, setting OKRs, empowering teams). It also mentions other related metrics like Product MRs by Type and Say Do Ratio.
  51. Want to iterate faster? Choose boring solutionsThis post advocates for the 'boring solutions' philosophy in software development, emphasizing the choice of established, popular technologies over novel or complex ones to reduce overall system complexity and enable faster iteration. It provides several examples from GitLab's history, including using issue labels for issue boards, using `curl` against API endpoints instead of building new UIs, using JSON Web Tokens for authentication, and using SQL for the CI job queue. It also mentions the migration from Gitolite to GitLab Shell as an example of abandoning a 'boring' solution when it no longer worked.
  52. Why iteration helps increase the merge request rateThis post introduces the concept of iterative development, inspired by John Boyd's iteration law, as a means to increase the merge request rate. It emphasizes a bias for action, accepting that 'all code is bad code' and embracing course correction, and the importance of writing things down and tightening feedback loops to foster a team environment that allows for rapid iteration and continuous improvement.
  53. How GitLab's QA Team Leverages Performance Testing ToolsThis post introduces the GitLab Performance tool (GPT), a new open-source tool built on k6, designed for performance testing of GitLab environments. It details the setup of reference architectures and test data, the capabilities of GPT (broad test suite, customization options, sequential/selective test runs, enhanced reporting, built-in success thresholds), and how GPT is used in automated CI pipelines for daily/weekly feedback and release cycle comparisons. It also encourages external users to adopt GPT for their own environments.
  54. One simple trick to make your screenshots 80% smallerThis post details a method to significantly reduce the file size of screenshots through color quantization (using pngquant) and DEFLATE compression (using zopfli). It provides command-line instructions and automation scripts for macOS using Hazel and Automator, demonstrating how to achieve up to 80% file size reduction. The technical details include explanations of PNG-32 vs. PNG-8, color quantization, and the benefits of zopfli over standard zlib.
  55. Why GitLab.com is changing its CDN provider to Cloudflare March 28GitLab.com is changing its CDN provider from Fastly to Cloudflare to improve availability, security, and performance. This involves integrating Cloudflare's WAF, Spectrum, and Argo. The change requires handling both HTTP and SSH traffic (ports 22 and 443) and necessitates firewall reconfiguration for users whitelisting GitLab.com IPs. Custom runner images or private runners with DNS or SSL certificate caching may also be affected.
  56. How to optimize your value exchange systemThis post introduces a framework for optimizing the value exchange system by shifting focus from output to outcomes, analyzing value streams, and applying continuous improvement principles (Kaizen) to identify and address bottlenecks in the R&D process. It uses a case study of Acme Co. to illustrate how a decline in revenue growth was traced back to slow capability delivery from R&D, which was a result of misaligned metrics (focusing on output over outcomes) and a lack of understanding of the value delivery stream. The post advocates for measuring value delivery streams and applying the Improvement Kata to achieve target conditions.
  57. Optimizing The Value Exchange: Reduce Waste To Increase FlowThis post details the identification and reduction of waste in the R&D value stream, specifically focusing on reducing 'waiting' time through process improvements like pair programming and work-in-progress limits, leading to a significant reduction in lead time. It also highlights the critical impact of technical debt and defects on the delivery of new value, leading to a strategic shift to address these issues.
  58. Shorter feedback loops compound value in developmentThis post introduces a new approach to optimizing the value exchange system by adopting throughput accounting and an intelligent decision support framework using NLP to extract topics and sentiment from text-based data, enabling better prioritization of the roadmap and a shift from measuring output to measuring outcomes through continuous innovation and lean sprints.
  59. Improve your productivity by tracking your time and measuring your E-factorThis post details the author's personal journey and strategies for improving productivity by increasing 'deep work' and 'E-factor' (uninterrupted hours / body-present hours). It introduces concepts from 'Deep Work' and 'Peopleware', and describes personal workflow adjustments (email timing, self-improvement activities, routine) and environmental factors (separate room, routine, timezone, Slack/email policy, task journaling, asynchronous work) that enabled these improvements. The author shares a Google Spreadsheet template for time tracking and a GitLab project for task journaling as tools to support these practices.
  60. A framework for sssessing data organization maturityThis post introduces a framework for assessing data organization maturity, categorizing data analysis into three tiers: reporting, insights, and predictions. It emphasizes that a mature data organization is foundational to a mature analytics organization. The post details the evolution of data analysis from gatekept access to democratized data, highlighting the value of insights derived from combining data sources. It also provides guidance on maturing data teams through investment, empowerment, and leveraging open-source analytics tools like dbt for speed to value and best practices.
  61. Store and update namespace statistics in a performant mannerIntroduced an asynchronous approach to update namespace storage statistics using Sidekiq jobs and Redis for lease management. This avoids increasing the transaction length of project statistics updates and improves performance for large namespaces. The solution involves creating a `namespace_aggregation_schedules` table, enqueuing Sidekiq jobs for immediate and delayed execution, and using Redis leases to prevent concurrent aggregations. A cron job is also scheduled to process any remaining pending updates.
  62. 6 Lessons we learned when debugging a scaling problem on GitLab.comThis post details the debugging of intermittent Git pull errors on GitLab.com caused by SSH connection drops. The investigation identified that scheduled jobs, running at the top of the minute, were overwhelming the `MaxStartups` limit on SSHD servers, leading to connection drops. The solution involved increasing the `MaxStartups` setting and implementing HAProxy rate limiting to gracefully delay connections instead of dropping them. Lessons learned include the utility of Wireshark's statistics menu, the importance of logging at default levels, and the impact of time synchronization on system load.
  63. We've open sourced 3 tools to help troubleshoot system performanceThis post introduces three open-source tools created by the GitLab Self-Managed Support team to aggregate and summarize performance data for self-managed customers: strace parser, JSON Stats, and GitLabSOS. The strace parser is detailed as a tool to summarize strace output, making it easier to identify performance bottlenecks by highlighting slow file operations and process execution times. An example demonstrates how strace-parser helped diagnose a slow filesystem issue impacting Gitaly.
  64. Lessons from implementing global code search on GitLab.comThis post details efforts to improve Elasticsearch integration for global code search on GitLab.com. Key technical challenges and attempted solutions include reducing Elasticsearch index size by investigating force merges (abandoned due to performance degradation), adjusting Edge NGram sizes for blobs (potential 7-15% savings, pending validation), and exploring separate indexes (no space savings but improved indexing speed). Administration capabilities are being enhanced with better cluster visibility, graceful recovery from indexing failures, selective/progressive indexing, making code indexing optional, and sharding Elasticsearch per group.
  65. 10 strategies for cycle time reductionThis post details ten strategies for reducing development cycle times, drawing from the experiences of GitLab engineering leads. Key strategies include automating work (CI/CD, manual tasks), breaking work into smaller iterative pieces, utilizing feature flags, measuring the impact of workflow components, improving cross-team communication and automation, and managing asynchronous work by having multiple items to work on. It also touches upon challenges like getting it right the first time, differentiating helpful from unhelpful processes, working across teams, and asynchronous communication.
  66. How our production team runs the weekly on-call handoverThe production team developed an 'on-call robot assistant' program that automates the generation of on-call handover reports. This program pulls data from PagerDuty (incidents), GitLab issues (production backlog), and GitLab Prometheus (key performance metrics) to create a GitLab issue for the handover meeting. This automates tedious work and ensures consistency in reviewing incidents and metrics.
  67. How a fix in Go 1.9 sped up our Gitaly service by 30xThis post details the debugging and resolution of performance degradation in the Gitaly service. The issue was identified as increasing latency over time, eventually leading to lockups. Through profiling (pprof) and cAdvisor monitoring, contention on `syscall.ForkLock` was pinpointed. Research led to the discovery that Go 1.9's switch from `fork/exec` to `posix_spawn` resolved this by reducing the overhead of process creation, especially with large virtual memory footprints. Recompiling Gitaly with Go 1.9 resulted in a 30x reduction in latency.
  68. Scaling the GitLab databaseThis post details the implementation of connection pooling using pgbouncer to address high database connection loads on GitLab.com. It explains the problems with direct PostgreSQL connections, the benefits of connection pooling, and the decision-making process between pgbouncer and pgpool-II. The post highlights the configuration of pgbouncer for transaction pooling, the reduction in `max_connections`, and the setup of separate pools for web requests and background jobs. It also discusses the side effect of disabling prepared statements and the resulting memory savings.
  69. GitLab Infrastructure UpdateThis post details the challenges and solutions related to scaling GitLab.com's infrastructure, specifically addressing storage issues with CephFS and PostgreSQL vacuuming performance under high load. The company is committed to documenting these solutions to help other organizations facing similar scaling challenges.
  70. Feature Highlight: Cycle AnalyticsIntroduced Cycle Analytics to measure and visualize the time it takes for a team to go from an idea to production. This feature breaks down the development process into distinct stages: Issue, Plan, Code, Test, Review, Staging, and Production. It relies on GitLab CI configuration and specific commit message patterns (e.g., 'Closes #xxx') to track progress through these stages. The initial iteration focuses on measuring median times for each stage and the total cycle time.
  71. Why we switched our philosophy from Ops to InfrastructureThis post details a philosophical shift from a reactive 'Operations mindset' to a proactive 'Infrastructure mindset' for managing GitLab's performance and scalability. Key steps include focusing on infrastructure from a development perspective, spotting patterns through time-based metrics, understanding performance gaps, aligning resources effectively, automating manual tasks, and iterating on solutions. It also describes a specific win in improving SSH Git access time by debugging and resolving issues related to `authorized_keys` file linear search, worker restarts, HTTP request queuing, and D-Bus connection limits. The post also lists the toolbox used for these efforts, including Chef, Prometheus, Grafana, ELK stack, and Sentry.
  72. Making GitLab FasterThis post details performance optimizations made in GitLab 8.5, specifically focusing on reducing the latency of fetching issues closed by a merge request and improving the performance of retrieving last update times for events. The optimization for fetching issues involved concatenating commit messages before processing, reducing database queries. The improvement for event update times involved optimizing the SQL query for Atom feeds by refining the logic for determining user project access.
  73. How GitLab uses Unicorn and unicorn-worker-killerThis post details GitLab's use of Unicorn as a pre-forking Ruby web server for handling web requests and Git HTTP clients. It explains Unicorn's multi-process architecture for CPU utilization and fault tolerance, including how the master process spawns and replaces worker processes that crash or time out. It also introduces the unicorn-worker-killer gem, a solution implemented to manage memory leaks in Unicorn workers by self-checking memory usage after a set number of requests and exiting if a threshold is exceeded, allowing the Unicorn master to replace the worker without dropping user requests. The post provides log examples of both worker timeouts and memory limit exits.