Edge Network Request Flow

11 posts · May 2016 – Aug 2023

What this trail covers

This thread tracks the evolution of Slack's edge network and request handling capabilities. Initial posts detail the foundational architecture for routing user requests from globally distributed edge PoPs to Slack's main region in AWS us-east-1. This includes the use of DNS resolution strategies with Route53 and NS1, Network Load Balancers (NLBs) for traffic ingestion, and the distinction between websocket and non-websocket traffic flows. This post details the implementation of a system to analyze real-time control protocol (RTCP) feedback from users to generate accurate feedback about individual uplink/downlink quality in Slack Calls, leveraging Janus and WebRTC.

All Slack trails

Aug 2023Traffic 101: Packets Mostly FlowThis post details the foundational architecture of Slack's edge network request flow. It explains how user requests are routed from globally distributed edge PoPs to the main AWS us-east-1 region. Key technical details include the DNS resolution process using Route53 and NS1 with EDNS Client Subnet (ECS), the role of Network Load Balancers (NLBs) in pass-through mode, and the separation of websocket traffic (ingested via 'envoy-wss') from non-websocket API traffic (ingested via 'envoy-edge'). It also describes failover mechanisms for websocket connections using primary and backup DNS records and the concept of a degraded mode for clients when websocket connections fail. For API traffic, it outlines routing to services like 'Supra/Miata', 'Imgproxy', and 'Envoy-www'.
Aug 2023Slack’s Migration to a Cellular ArchitectureThis post details the migration of Slack's critical user-facing services from a monolithic to a cell-based architecture over 1.5 years. The motivation was to increase redundancy and limit the blast radius of site failures, particularly after an incident where a network disruption in one Availability Zone (AZ) caused user-visible errors. The solution involves treating AZs as cells, where services are siloed within their AZ and only communicate with services in the same AZ. This containment allows traffic to be dynamically routed away from failing AZs by reweighting target clusters in edge Envoy load balancers via Rotor, Slack's xDS control plane. This implementation satisfies goals of rapid propagation (seconds), graceful drains, incremental traffic assignment (1% granularity), and resilience against AZ failure.
Apr 2023Real-time MessagingThis post details the architecture for real-time messaging at Slack, focusing on the core services: Channel Servers (stateful, in-memory, serving subsets of channels via consistent hashing), Gateway Servers (stateful, in-memory, multi-region, client interface), Admin Servers (stateless, in-memory, webapp to CS interface), and Presence Servers (in-memory, user online status). It describes the client websocket setup, message broadcasting flow from client to Webapp, AS, CS, and then to multiple GSs and clients, and the handling of events, including transient events like typing indicators. The system achieves linear scalability to serve tens of millions of channels per host and tens of millions of connected clients, delivering messages globally within 500ms.
Jan 2023What We Learned from Building GovSlackThis post details the challenges and solutions for building and operating infrastructure within AWS GovCloud for GovSlack. It covers the account relationship between GovCloud and commercial accounts, the process of creating GovCloud accounts, and the strict isolation required between GovDev and GovProd environments using separate AWS organizations and a shared-nothing paradigm. It highlights specific challenges such as the lack of CloudFront and public Route53 zones in GovCloud, the need for the `--region` flag with the AWS CLI, and strategies for managing ACM certificates and transferring artifacts between partitions. The post also discusses adapting Terraform modules to handle GovCloud ARNs.
Nov 2021The Case of the Recursive ResolversThis post details the challenges and root cause analysis of a 24-hour outage caused by an attempt to enable DNSSEC on slack.com. It explains the DNS protocol, DNSSEC, and Slack's DNS infrastructure. The post describes three unsuccessful attempts to enable DNSSEC, detailing the specific issues encountered in each, including ISP outages, VPN resolver strictness, and recursive resolver caching behavior leading to NODATA responses. The incident highlights the complexities of DNSSEC rollouts and the importance of understanding resolver caching mechanisms.
Advertisement
Oct 2021Building the Next Evolution of Cloud Networks at Slack – A RetrospectiveThis post is a retrospective on the Whitecastle network redesign project, detailing lessons learned and improvements made approximately one year after its initial rollout. Key learnings include the complexities of proxy environment variables across different languages, the critical importance of Transit Gateway visibility and scaling, and the process of migrating from VPC peering to Transit Gateways. Improvements highlighted are the adoption of multiple workload VPCs per region for flexibility and the management of private Route53 zones across different environments.
Mar 2021Migrating Millions of Concurrent Websockets to EnvoyThis post details the migration of Slack's websocket load balancing tier from HAProxy to Envoy Proxy. It outlines the motivations for the migration, including Envoy's hot restart capabilities and advanced load balancing features (zone-aware routing, outlier detection, panic routing). The post describes the process of generating Envoy configurations using Chef libraries and custom resources, the testing and validation methods employed (curl, debug logging, admin interface, JSON logs), and the production rollout strategy using weighted DNS routing via NS1 and NLBs. The migration aimed to improve operability, standardize on a single proxy technology, and leverage Envoy's advanced features.
Jun 2020A Terrible, Horrible, No-Good, Very Bad Day at SlackThis post details the root cause of the May 12th, 2020 outage, which was a bug in the `haproxy-server-state-management` program that led to stale HAProxy backend configurations. The bug occurred because the program attempted to find slots for new webapp instances before freeing slots for old ones, causing it to fail and exit early when no slots were available. This resulted in HAProxy instances not receiving updated backend lists, especially during traffic scaling events. The outage was mitigated by a rolling restart of the HAProxy fleet. The post also highlights the ongoing migration from HAProxy to Envoy Proxy for ingress load balancing, noting that Envoy's native integration with an xDS control plane will prevent similar issues.
May 2017Flannel: An Application-Level Edge Cache to Make Slack ScaleIntroduced Flannel, an application-level edge cache deployed at Slack's edge points-of-presence. Flannel caches relevant data (users, channels, bots) upon client startup and provides query APIs for on-demand fetching. It proactively pushes data to clients, foreseeing future requests. This architecture reduces client startup time, memory footprint, and the impact of reconnect storms by serving slimmed-down startup data and handling client queries at the edge. It uses consistent hashing for team affinity and serves 4 million simultaneous connections, handling 600K client queries per second.
Jun 2016Making Slack Feel Like SlackThis post details the introduction of libslack, a C++ library designed to consolidate client-side logic for caching, syncing, and hiding latency. It describes how libslack is being integrated into mobile (Android, iOS) and desktop clients, and how a server version of libslack is being deployed in edge POPs to act as an application-aware cache. This enables a unified communication protocol between clients and the edge, and between the edge and the mothership, improving caching strategies and reducing round-trip times. The post also discusses the challenges of managing diverse client codebases and the benefits of strong typing and asynchronous APIs provided by libslack.
May 2016Calls: Is it you or is it me?This post details the implementation of a system to analyze real-time control protocol (RTCP) feedback from users to generate accurate feedback about individual uplink/downlink quality in Slack Calls. It explains how to analyze RTCP Receiver Reports (RR) to calculate download loss by determining the realistic loss on each downlink leg, and how to calculate upload loss by tracking RTP packet sequence numbers. The system uses Janus as the selective forwarding unit (SFU) and WebRTC for voice chat, adding custom support for analyzing RTCP feedback to pinpoint network issues.