TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Anomaly Event Response

5 posts · Jun 2020 – Sep 2025

What this trail covers

This thread tracks the evolution of Slack's Anomaly Event Response (AER) system, a proactive defense mechanism designed to automatically identify and respond to suspicious user activities in real-time. Initial posts detail the foundational architecture, detection mechanisms, and response capabilities of AER, focusing on compressing the detection-to-response window from days/hours to minutes. This post details the system's design philosophy, focusing on common threat types like Tor access, excessive downloads, and unusual user agents, and provides guidance on correlating and aggregating anomalies for deeper security insights. It also elaborates on the UI for investigating audit logs and anomalies, and the ability to terminate user sessions.

All Slack trails
  1. Building Slack’s Anomaly Event ResponseThis post details the design philosophy of AER, prioritizing common threat types such as Tor access, excessive downloading, data scraping, session fingerprint mismatches, and unexpected API volumes. It elaborates on the system architecture, including the detection engine's use of rule-based heuristics and dynamic thresholds calibrated to enterprise usage patterns, the decision framework's validation of anomalies against customer configurations and internal rules, and the response orchestrator's session termination, audit log generation, and configurable notification mechanisms. It also highlights key findings and impact, emphasizing the reduction in data breach response time and cost.
  2. Slack Audit Logs and AnomaliesThis post elaborates on the Anomaly Event Response (AER) system by detailing the nature of anomaly events within audit logs, distinguishing them from regular audit events. It provides practical guidance on how organizations can operationalize anomalies, emphasizing that they are indicators for further investigation rather than direct incident triggers. Specific strategies for allowlisting CIDR ranges and ASNs, correlating multiple anomalies (e.g., user_agent with excessive_downloads), and aggregating anomalies over time are discussed. The post also highlights the importance of consuming full audit logs for context when investigating anomalies and provides a detailed walkthrough of the Audit Log UI for security detections and event details, including terminating user sessions.
  3. Slack’s Incident on 2-22-22This post details a major incident on 2-22-22 that exposed cascading failure scenarios within Slack's infrastructure. It highlights the complex interactions between the client boot process, Vitess datastores, caching (Memcached/Mcrouter/Mcrib), and service discovery (Consul). The incident was triggered by Consul agent restarts impacting Mcrib's ability to maintain cache stability, leading to cache misses and subsequent database overload due to inefficient scatter queries for GDM membership. Mitigations involved aggressive throttling of client boot requests and a more granular approach to increasing limits. The post explains the tipping point where cache misses lead to database overload and prevent cache refilling, emphasizing the need for robust cache management and efficient data querying strategies.
  4. Slack’s Outage on January 4th 2021This post details the incident response to Slack's outage on January 4th, 2021. It describes the root cause as network saturation in AWS Transit Gateways, exacerbated by a post-holiday traffic surge. The incident revealed dependencies of monitoring services on the same network infrastructure, leading to a loss of visibility during the outage. It also details issues with autoscaling and instance provisioning under degraded network conditions, specifically the overloading of the 'provision-service' due to increased latency and resource usage. The post outlines the steps taken to recover, including disabling downscaling, manual upscaling attempts, and the eventual recovery as AWS increased TGW capacity. Post-incident actions include architectural changes to isolate monitoring services, preemptive upscaling of TGWs, and reevaluation of health-checking and autoscaling configurations, along with load testing of critical services.
  5. All Hands on DeckThis post details the incident response process during a major outage on May 12th, 2020, highlighting the 'All Hands on Deck' approach, the transition from a heroic on-call model to a FEMA-inspired Incident Command System, and the specific steps taken to diagnose and mitigate a HAProxy configuration issue that led to a full service disruption. It emphasizes the importance of structured incident response, learning from failures, and implementing action items to prevent recurrence.