Defense System Lifecycle Management

7 posts · Jul 2016 – Jan 2026

What this trail covers

This feature thread tracks the evolution of managing defense systems, including rate limits and traffic controls, to ensure they remain effective and do not inadvertently block legitimate users. Initially, the focus was on building these defenses. This post details the development and open-sourcing of synsanity, an iptables module for Linux 3.x that mitigates SYN flood attacks by generating SYN cookies outside the kernel's LISTEN socket lock, allowing for multi-processor utilization during attac. The thread now also covers the lifecycle management of these protections, emphasizing their temporary nature, the need for active maintenance, and improved observability to prevent outdated rules from blocking legitimate traffic.

All GitHub trails

Jan 2026When protections outlive their purpose: A lesson on managing defense systems at scaleThis post details the discovery and remediation of outdated protection rules that were incorrectly blocking legitimate users. It highlights the challenge of managing the lifecycle of emergency incident mitigations, which can become technical debt if not actively reviewed and removed. The post emphasizes the need for better visibility across protection layers, treating incident mitigations as temporary by default, and implementing post-incident practices to evaluate and evolve emergency controls into sustainable solutions. It also provides a simplified diagram of request flow through defense layers and a lifecycle diagram for incident mitigations.
Apr 2021How we scaled the GitHub API with a sharded, replicated rate limiter in RedisThis post details the migration of GitHub's rate limiter from Memcached to a sharded, replicated Redis backend. It describes the challenges with the old system (Memcached persistence, datacenter routing) and the proposed solution using Redis with client-side sharding, Lua scripts for atomicity, and Redis expiration. It also details two critical bugs encountered post-release: 'reset at' header wobbling due to clock drift between Redis and Ruby, and incorrect rate limit rejections caused by stale data read from Redis replicas. Fixes involved managing 'reset at' in application code and ensuring responses accounted for replica reads and potential expirations.
Sep 2020How we threat modelThis post details the process and benefits of threat modeling at GitHub. It explains what threat modeling is, outlines the process (deciding when to threat model, building the model using tools like Microsoft's Threat Modeling Tool or OWASP's Threat Dragon, and reviewing the model using frameworks like STRIDE), and highlights key learnings such as system-wide security improvements, proactive design guidance, and improved communication between security and engineering teams. It emphasizes merging threat modeling into existing development lifecycle processes, ensuring preparedness, using defined mechanisms for risk assessment, and leaving with specific action items.
Oct 2017Mitigating replication lag and reducing read load with frenoThis post introduces 'freno', a new centralized throttling service designed to replace existing, disparate throttling mechanisms. freno addresses operational issues and limitations by continuously and asynchronously polling MySQL replicas for replication lag. It provides a unified API for applications to query lag status and receive recommendations, enabling better control and visibility over large write operations. The post details the architecture of freno, including its continuous polling mechanism and its role in managing replication lag for background jobs and schema migrations.
Feb 2017A glimpse into GitHub's Bug Bounty workflowThis post details the internal workflow for triaging and managing bug bounty submissions, including initial contact, response actions, risk assessment, fixing the issue, and closing the loop with researchers. It also mentions the release of a HackerOne API client library to improve automation within bug bounty programs.
Advertisement
Jan 2017Moving persistent data out of RedisThis post details the migration of persistent Redis usage to MySQL's GitHub::KV key/value store, specifically addressing the challenges of migrating activity feeds. It describes optimizing write operations for fanout scenarios and the eventual migration of event feeds to MySQL, improving performance and reducing operational costs.
Jul 2016SYN Flood Mitigation with synsanityIntroduced synsanity, an iptables module for Linux 3.x, to mitigate SYN flood attacks. synsanity generates SYN cookies outside the LISTEN socket lock, enabling multi-processor utilization during attacks. It validates the ACK packet and then allows it through to the standard Linux SYN cookie code, without touching subsequent packets. This avoids the need for a custom kernel and allows for maintenance of existing, patched kernels. The module is open-sourced.