TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

EC2 Platform Evolution

12 posts · Sep 2020 – Jul 2026

What this trail covers

This thread tracks the evolution of Slack's infrastructure management, focusing on the use of Terraform and network architecture. Initial posts detailed foundational architecture and challenges with single AWS accounts. This post details the "Shipyard" platform, a next-generation EC2 platform that treats infrastructure as deployable artifacts rather than mutable instances. It brings modern deployment practices like immutability, progressive rollouts, and automated safety directly to EC2 instances, supporting multi-architecture and multi-OS, metrics-driven deployments with safety controls, fast and predictable provisioning via layered images, simplified configuration management by shifting from continuous enforcement to image baking and provisioning, real-time inventory and fleet visibility via the Peekaboo system, and short-lived, continuously refreshed instances. The platform uses a shared "slack-zero" base image built with AWS Image Builder and service-specific images built on top, separating instance preparation into baking and provisioning phases. Deployments are handled through controlled replacements using AWS Instance Refresh and Karpenter, with emergency fixes managed via AWS Systems Manager.

All Slack trails
  1. Shipyard: How We Built Slack’s Next-Generation EC2 PlatformThis post introduces Shipyard, Slack's next-generation EC2 platform, which shifts from continuous configuration management to an immutable, artifact-based approach for EC2 instances. Key contributions include: the design principles of Shipyard (immutability, progressive rollouts, automated safety), multi-architecture/OS support, metrics-driven deployments with safety controls integrated with Gondola, fast provisioning via layered images (slack-zero base image + service images), simplified configuration management (baked into images, applied during provisioning), real-time fleet visibility via the Peekaboo inventory system (using AWS EventBridge, OpenSearch, Lambda), and short-lived, continuously refreshed instances. It details the use of AWS Image Builder for slack-zero base image creation and the separation of instance preparation into 'baking' and 'provisioning' phases. Deployment strategies using AWS Instance Refresh and Karpenter are also covered.
  2. Agentic Testing: Where Agents Fit in the E2E Testing StackThis post details the integration of agentic testing into the E2E testing stack, exploring how agents can be utilized within the Shipyard platform for more robust and efficient testing.
  3. Slack AI: The Path to Multi-CloudThis post details the foundational build of the Shipyard platform, outlining its architecture, the technical decisions made, and the challenges overcome in its creation. It also details the integration of scalable network probing and HTTP/3 readiness using Prometheus within Shipyard, and discusses the challenges and strategies for operating Shipyard in a multi-cloud environment, including considerations for network latency, data locality, and cost optimization.
  4. From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data PipelinesThis post details the modernization of Slack's EMR data pipelines, moving from SSH to REST. It focuses on security-driven improvements and the integration with Shipyard, Slack's next-generation EC2 platform. The post outlines the architectural changes, the security considerations addressed, and the technical challenges overcome in this migration, including the use of Prometheus for network probing and HTTP/3 readiness.
  5. Managing context in long-run agentic applicationsThis post contributes by detailing the challenges and solutions for managing context in long-running agentic applications. This is relevant to the EC2 platform evolution as agentic workflows are likely to be deployed and managed on this infrastructure, requiring efficient context management for their successful operation and scalability.
  7. From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with PrometheusThis post details the integration of scalable network probing and HTTP/3 readiness using Prometheus within Slack's Shipyard EC2 platform, building upon the foundational architecture previously described.
  8. How Slack Rebuilt Notifications 📣This post details the foundational build of the Shipyard platform, outlining its architecture, the technical decisions made, and the challenges overcome in its creation. It also details the integration of scalable network probing and HTTP/3 readiness using Prometheus within Shipyard.
  9. Streamlining Security Investigations with AgentsThis post demonstrates how the Shipyard EC2 platform is being utilized to streamline security investigations through the implementation of agents. It highlights a practical application and advancement of the platform's capabilities beyond its initial architectural description.
  10. Advancing Our Chef Infrastructure: Safety Without DisruptionIntroduced a split production Chef environment strategy, dividing a single environment into six isolated "buckets" (`prod-1` to `prod-6`) mapped to Availability Zones. Extended the `Poptart Bootstrap` tool to assign nodes to these specific environments. Implemented a phased rollout process for cookbook updates, with `prod-1` acting as a canary and subsequent environments (`prod-2` to `prod-6`) following a release train model, ensuring changes are tested in production incrementally and reducing the risk of widespread impact from faulty deployments.
  11. Advancing Our Chef InfrastructureThis post details the transition from a single Chef stack to a sharded Chef infrastructure at Slack. Key technical contributions include: 1. Implementing AWS Route53 Weighted CNAME records for directing new provisions to specific Chef shards. 2. Separating development and production Chef infrastructure into distinct stacks. 3. Leveraging Consul for service discovery to replace Chef searches, overcoming a circular dependency with Nebula using `ruby_block` resources and the `lazy` keyword for attribute assignment. 4. Developing 'Shearch' (Sharded Chef Search) API to query across multiple shards and 'Gnife' (Go Knife) for sharded command-line operations. 5. Modifying the cookbook upload process by deploying multiple DishPig instances and using SNS/SQS for fan-out notifications to individual Chef stacks.
  12. Our Journey Migrating to AWS IMDSv2This post details Slack's migration of its EC2 fleet and tools to AWS IMDSv2 to enhance security by mitigating SSRF vulnerabilities inherent in IMDSv1. It outlines the discovery process using the `MetadataNoToken` CloudWatch metric and the `imds-cw-metric-collector` application, and the investigation of IMDSv1 call origins using `AWS ImdsPacketAnalyzer` and other tools. The post describes the migration of services and scripts to IMDSv2-compatible libraries and the implementation of preventative measures for new instances through Terraform modules (`accounts_using_imdsv1`) and AWS Service Control Policies (SCPs) to enforce IMDSv2. It also highlights the limitations of SCPs and the establishment of an EventBridge and Lambda-based notification system as a final safety net.
  13. How We Use Terraform At SlackThis post details the evolution of Slack's Terraform state file management, including the transition from a single state file to a distributed model with dedicated accounts for teams and services. It describes the use of Jenkins pipelines for deployment, the shift in ownership of Terraform states and pipelines to service owners, and the storage of state files in S3 with DynamoDB for locking. The post also covers the creation of 'Ops' boxes for developer testing and prototyping, and the strategies for managing multiple Terraform and provider versions, including the development of a Golang tool for upgrade management.
  14. Building the Next Evolution of Cloud Networks at SlackThis post details the "Whitecastle" project, a network architecture redesign at Slack. It introduces the use of AWS Shared VPCs to overcome AWS rate-limiting issues and manage IP spaces across multiple child accounts. It also describes the implementation of AWS Transit Gateway Inter-Region Peering to enable communication between different AWS regions, with an automated Terraform-based approach for managing peering connections and route tables. A "Whitecastle Network Tester" application is introduced for real-time network connectivity validation and monitoring. Finally, it outlines an incremental migration strategy for services to adopt the new network architecture.