TechPrufer
Get started
GitHub
GitHub logoGitHub · Feature trail

Platform Problem Solving and Domain Understanding

5 posts · May 2017 – Jun 2025

What this trail covers

This feature thread tracks the evolution of how engineers tackle platform problems, emphasizing domain understanding, bridging concepts to platform-specific skills (networks, OS, IaC, distributed systems), knowledge sharing, understanding impact radius, and robust testing strategies for platform changes. It highlights the shift from product engineering to platform engineering and the unique challenges and best practices associated with the latter, particularly concerning services like DNS. The post details the importance of understanding domain, bridging concepts to platform-specific skills like networking, OS, IaC, and distributed systems, and emphasizes knowledge sharing. It also delves into the impact radius of platform changes and robust testing strategies, including IaC, E2E testing, and self-healing mechanisms.

All GitHub trails
  1. How GitHub engineers tackle platform problemsThis post details best practices for platform engineers tackling complex problems. It emphasizes understanding the domain through communication and documentation, bridging concepts to platform-specific skills such as networking (TCP, UDP, L4 load balancing, dig), operating systems, Infrastructure as Code (Terraform, Ansible, Consul), and distributed systems (failover, recovery). It highlights the importance of knowledge sharing for collaboration and preventing knowledge loss. The post also discusses the impact radius of platform changes, particularly for foundational services like DNS, and the necessity of robust testing strategies including IaC, end-to-end testing, and self-healing mechanisms.
  2. How we organize and get things done with SERVICEOWNERSThis post introduces SERVICEOWNERS as a new layer of abstraction for organizing engineers and code around services within GitHub's large Ruby on Rails monolith. It details the `SERVICEOWNERS` file, `service-mappings.yaml`, the `serviceowners` Ruby gem, and the `ownership.yaml` file. The system aims to provide consistent, company-wide grouping of functionality to humans, leading to a shared lexicon, reduced disruption from team reorganizations, and a centralized service catalog. The service catalog visualizes service relationships, dependencies, and information about maintainers and SLAs, which aids in onboarding and incident remediation. The gem generates the `CODEOWNERS` file dynamically and provides a chat-op for querying service ownership.
  3. Introducing the new GitHub Status SiteIntroduced a new GitHub Status Site that provides granular component status updates (e.g., Git operations, API requests, Pages builds) and allows users to subscribe to these updates via email, SMS, or webhooks. This improves incident communication accuracy and enables integration with operational tools like Jenkins for automated backoff mechanisms. The old status site and its API will be deprecated in favor of the new system.
  4. GLB: GitHub's open source load balancerThis post details the design and open-sourcing of GLB, GitHub's Layer 4 load balancer. It explains how GLB uses ECMP routing and rendezvous hashing to scale a single IP address across multiple physical machines while minimizing connection disruption during server changes. The GLB Director tier is stateless, forwarding packets to primary and secondary servers based on a pre-computed forwarding table, which provides a 'second chance' mechanism for connections. The post also contrasts GLB's approach with traditional LVS director designs and discusses the invariants maintained by rendezvous hashing for consistent server selection.
  5. DNS Infrastructure at GitHubThis post details the design and implementation of a new DNS infrastructure at GitHub to address limitations of the previous system, including scalability, visibility, and availability issues. The new architecture introduces a tiered design with isolated roles: caches (recursive resolvers), edges (DNS authorities for zone transfers), and authorities (hidden masters storing data in MySQL). It leverages Anycast for cache accessibility, regional edge hosts for local zone copies, and an HTTP API for record management. The post also highlights the selection of Unbound, NSD, and PowerDNS for their metric capabilities and discusses the operability improvements, including enhanced observability and resilience against connectivity issues.