Application Security Program Development

2 posts · Apr 2019 – Sep 2026

What this trail covers

Shopify has evolved its application security program by developing and enhancing an agentic code review and test oracle harness named Dispatch. This system uses multiple AI agents to discover vulnerabilities in software, prove them with real tests, and provide Shopify-tuned fixes. The harness is designed to be durable and outlast individual models, focusing on integrating with existing testing pipelines and producing actionable, context-specific pull requests for developers. It has been applied to large-scale applications, demonstrating cost-effectiveness and the ability to identify critical findings. The system emphasizes deterministic scripting for reliable outputs and robust partitioning strategies to optimize cost and accuracy.

All Shopify trails

Sep 2026How River takes security work from a fix to merge (2026) - ShopifyThis post details the operationalization of an AI agent named River for automating security vulnerability remediation. River handles dependency upgrades and first-party application vulnerability fixes by revalidating findings against live code, repairing mechanical failures, rebasing PRs, retriggering CI, and handing off decisions requiring human judgment. It significantly reduced the backlog of open security issues and increased the rate of security merges. Key contributions include the agent's ability to revalidate findings against live repository state, repair mechanical failures caused by upgrades, and verify closure by checking repository head and tracker state after merges. The post emphasizes the importance of revalidating before editing, ensuring evidence follows code, and making handoff a deliberate part of the workflow.
Jul 2026Building an agentic harness that outlasts the model (2026) - ShopifyThis post details the evolution of Shopify's agentic code review harness, Dispatch. It introduces a multi-agent orchestration workflow that includes test bootstrapping, architecture documentation generation, file cataloging, partitioning, parallel hunting with specialized agents, sequential verification using a different model, post-processing for deduplication and severity scoring, reporting, and automated remediation via PR generation. Key lessons learned include building test oracles for web vulnerabilities, producing consistent results through rigorous principles, keeping costs low via partitioning, and leveraging deterministic scripts. The post highlights the harness's durability over individual models and its application to large-scale systems like Shopify Core, yielding significant findings and cost savings.
Jun 2021Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time - ShopifyThis post details Shopify's transition to using Let's Encrypt as its primary Certificate Authority for issuing and renewing TLS certificates for merchant domains. It highlights the challenges of programmatic certificate management with previous providers, the benefits of Let's Encrypt's ACME protocol for automation, and the successful implementation of a system capable of provisioning and renewing certificates for millions of domains within hours, significantly improving responsiveness for emergency rotations.
Apr 2019Building Shopify’s Application Security Program - ShopifyThis post outlines the foundational principles and early strategies for building Shopify's application security program. It emphasizes standardization of the technology stack (Ruby on Rails), gaining visibility into services and dependencies, and the growth of the security team through specialization (fraud, appsec, infra, etc.). It also highlights the use of external validation like penetration testing and bug bounty programs, and the importance of fast, automated, and risk-aware security interactions. Specific tactics mentioned include renaming dangerous methods like `html_safe` to `dangerously_output_as_html` and triggering alerts for the appsec team. The post also touches on embedding security decisions at the infrastructure level.
Advertisement