GitHub.com Deployment System

14 posts · Jun 2015 – Jun 2024

What this trail covers

This feature thread tracks the evolution of GitHub.com's deployment system. Initially, the focus was on establishing a robust workflow for deploying code from pull request branches to production, incorporating automated checks and visibility through chat. This post details improvements to the orchestration framework by storing state history in a separate table for incremental reads/writes, reducing CPU consumption for orchestrations. It also details reducing postback load by only sending step data for completed steps. The overall goal is to improve the reliability and efficiency of push processing by decoupling the monolithic `RepositoryPushJob` into smaller, independent, and parallelized background jobs triggered by Kafka events.

All GitHub trails

Jun 2024How we improved push processing on GitHubThis post details the migration of GitHub's push processing from a monolithic `RepositoryPushJob` in the Ruby on Rails monolith to a decoupled, event-driven architecture using Kafka. The monolithic job was sequential, had a large blast radius, and suffered from conflicting retry requirements. The new architecture introduces a Kafka topic for push events, which are then consumed by independent background jobs, each responsible for a specific set of tasks (e.g., AppsOnPushJob, CodespacesOnPushJob, PullRequestsOnPushJob). This change reduces the blast radius of problems, lowers latency for push-related tasks (e.g., pull request sync time), improves observability, and increases the reliability of push processing from 99.897% to an estimated 99.999%. Investments were made in reliable Kafka publishing, dedicated job workers, improved observability, and a consistent per-event feature flagging system.
Mar 2024How GitHub uses merge queue to ship hundreds of changes every dayThis post details the transition from an internal 'trains' system for managing pull request deployments to a generalized merge queue. It explains the pain points of the 'trains' system, including its complexity, impact on developer satisfaction, and limitations on deployment size. The merge queue is presented as a solution that improves developer experience by simplifying the shipping intent, preventing problematic pull requests from impacting others, and providing a consistent, automated process across repositories. The post highlights the scale of the merge queue's usage at GitHub, including the number of pull requests merged and CI runs, and quantifies the improvements in shipping velocity and developer satisfaction.
Sep 2023How GitHub uses GitHub Actions and Actions larger runners to build and test GitHub.comThis post details the adoption of GitHub Actions and larger runners for GitHub.com's CI system. It highlights the benefits of autoscaling, managed infrastructure, and custom VM images for faster bootstrapping and pre-installation of tools. The post also discusses the implementation of reusable workflows to standardize CI across repositories and the feature for reusing previous workflow outcomes to bypass redundant CI checks. Additionally, it addresses the challenge of private service access from larger runners by detailing a remote access solution using OIDC tokens and an example gateway.
Aug 2023How we build containerized services at GitHub using GitHubThis post details GitHub's "paved path" for building containerized services using Kubernetes and Docker. It outlines the onboarding process for new services, including scaffolding with Hubot and GitHub Apps, generating Kubernetes manifests, and setting up CI builds. The deployment process is described, involving ChatOps commands, fetching Docker images, updating manifests, and applying them to clusters, with integration of merge queues and deployment pipelines. Security measures are detailed, such as pre-built Docker images, vulnerability scanning (Dependabot, code scanning, secret scanning), authentication/authorization, internal network access, branch protection, and centralized secret management. The end-to-end deployment flow from merge to rollout is illustrated with a diagram and step-by-step explanation.
Apr 2023Building GitHub with Ruby and RailsThis post details GitHub's strategy for weekly Ruby and Rails upgrades. It describes an automated GitHub Actions workflow that triggers a pull request to bump the Rails version to the latest commit daily, with all builds running on the new version. It also outlines a similar process for Ruby upgrades, testing alpha/beta versions in parallel builds and using CI to maintain compatibility. The post highlights the benefits of this approach, including faster upgrades, reduced patching, improved security, and easier contribution back to the community. It also emphasizes the importance of a thorough test suite, great test environments, and progressive rollout deploys for successful weekly upgrades.
Advertisement
Feb 2023Enabling branch deployments through IssueOps with GitHub ActionsThis post introduces the 'branch-deploy' GitHub Action, enabling IssueOps for branch deployments. It details the branch deploy model, contrasting it with the traditional deploy-merge model, and highlights its benefits for reliability. The post provides a detailed example of how to configure and use the action within a GitHub Actions workflow, including triggering noop and regular deployments via comments on pull requests. It also outlines the action's features such as configurable triggers, environment targets, and integration with GitHub deployments, and provides examples for various deployment targets.
Sep 2022How we tripled max concurrent jobs to boost performance of GitHub ActionsThis post details improvements to the GitHub Actions orchestration framework by changing the database schema to store state history in a separate table (`tbl_OrchestrationSessionEvent`) for incremental reads and writes, reducing CPU consumption for running orchestrations by 50% on average. It also details reducing postback load by stopping step status updates during workflow runs and only posting step data upon completion, and removing duplicated job completed events. These changes allowed GitHub Actions to run 3x more jobs concurrently on GHES machines.
Apr 2022Improving Git push times through faster server side hooksRewrote pre-receive hooks from Ruby to Go, reducing average execution time from 880ms to 10ms by eliminating dependency loading overhead. This significantly improves Git push performance for all users.
Apr 2022Performance at GitHub: deferring stats with rack.after_replyImplemented `rack.after_reply` functionality in Unicorn to defer expensive telemetry calls (statsd metrics flushing) out of the critical request path. This resulted in a 30ms p50 and 50ms+ p99 improvement on every request across the site. Developed a wrapper class for `rack.after_reply` to buffer and flush telemetry data after the response is sent. Added timeout behavior to prevent `rack.after_reply` code from running too long and increasing HTTP queueing.
Apr 20224 ways we use GitHub Actions to build GitHubThis post details improvements to the GitHub.com deployment system's orchestration framework by storing state history in a separate table for incremental reads/writes, reducing CPU consumption. It also details reducing postback load by only sending step data when necessary.
Jan 2022How we ship GitHub Mobile every weekThis post details the automation of the GitHub Mobile app's weekly release process using GitHub Actions and fastlane. It outlines the steps involved in generating release candidate builds, uploading to TestFlight, running tests, creating tracking issues, and preparing release notes. The automation covers most of the release pipeline, with manual intervention primarily for writing customer-friendly release notes and final app submission. The post highlights the use of GitHub Secrets for credentials, GitHub CLI for issue and PR creation, and PagerDuty integration for release engineer rotation. It also describes the version number management process via automated PRs and the overall weekly release timeline, emphasizing the benefits of reduced manual effort and improved onboarding.
Mar 2021How GitHub Actions renders large-scale logsThis post details the implementation of UI virtualization for rendering large-scale logs in GitHub Actions. It addresses limitations of previous libraries, such as fixed height requirements and issues with dynamic visibility changes. The team developed a custom virtualization library from scratch to render up to 50k log lines, support variable log line heights, implement sticky headers for steps, and ensure smooth scrolling and text selection. The approach involves estimating heights before rendering, optimizing DOM structure, and grouping log lines into clusters to reduce DOM mutations during scrolling.
Oct 2020Making GitHub CI workflow 3x fasterIntroduced a 'deferred compliance' tool to decouple long-running CI jobs (specifically integration tests for GitHub Enterprise Server) from the immediate requirement to pass before merging pull requests to GitHub.com. This tool creates a GitHub issue upon CI failure, tags the author and code owners, sends a Slack notification, and initiates a 72-hour timer for remediation. If the issue is not resolved within 72 hours, all deployments to GitHub.com are halted. This approach aims to improve Lead Time for Changes for GitHub.com deployments while maintaining accountability for Enterprise Server quality.
Jun 2015Deploying branches to GitHub.comThis post details the introduction of deploy queues to manage concurrent deployments, deploy guards to enforce CI/testing success and branch freshness, and deploy locks to prevent concurrent deployments. It also introduces various deployment environments, including a 'branch lab' for isolated experimental deployments.