TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Notification Tracing

5 posts · Jun 2017 – Mar 2026

What this trail covers

This thread tracks the evolution of Slack's notification system, focusing on improving completeness, reliability, and debuggability through end-to-end tracing. Initial posts detail the challenges of complex notification flows and the need for standardized data formats. This post details the foundational work to implement a tracing system for notifications, modeling each notification as a separate trace linked to the original message trace. This enables 100% sampling for notification flows, leading to a unified notification model across desktop and mobile with simplified preferences (All new posts, Mentions, or Mute) and decoupled push controls. The system now offers consistent state across clients, improved user control, and a significant reduction in support burden. Data shows a 5x increase in settings engagement and higher usage of push notification toggles, indicating users feel more in control and less overwhelmed.

All Slack trails
  1. How Slack Rebuilt Notifications 📣This post details the technical implementation of Slack's notification system rebuild, focusing on migrating from four conflicting preference models to a unified system with three clear channel notification options (All new posts, Mentions, or Mute) and decoupled push toggles for desktop and mobile. It describes the 'Preference Refactor' using a read-time strategy to migrate users without database changes, introducing `desktop_push_enabled` for push control. The 'Modal Makeover' introduced auto-save behavior and separated 'what' from 'how' notifications are received. 'Cross-Platform Parity' was achieved through reusable React components and refactoring legacy iOS code to match desktop structure. Lessons learned include the importance of trust in rollbacks, the impact of schema issues, and the clarity gained by removing sync parameters and storing explicit desktop/mobile values.
  2. Tracing NotificationsThis post details the foundational work to implement end-to-end tracing for Slack notifications. It addresses the challenges of complex notification flows and the need for 100% fidelity by modeling each notification as its own trace, linked to the original message trace using span links. This approach simplifies instrumentation, reduces storage and query overhead, and enables consistent data formatting and sessionization across backend and client systems. The post also outlines the specific spans and tags used to capture the notification lifecycle from trigger to read.
  3. Client Tracing: Understanding Mobile and Desktop Application Performance at ScaleThis post details the foundational work to implement client-side tracing for Slack's mobile and desktop applications (iOS, Android, JavaScript). It explains how distributed tracing concepts are applied to client performance analysis, breaking down API requests into queue time, HTTP request time, and parse time. It also details how database operations are traced, including wait time, execution time, and save time. The post introduces a custom Client Tracing API designed for client-specific needs and discusses production rollout strategies and sampling.
  4. Tracing at Slack: Thinking in Causal GraphsThis post introduces the foundational concepts and motivations behind Slack's new tracing system, which models traces as Causal Graphs composed of SpanEvents. It highlights the limitations of traditional distributed tracing systems (request-centric APIs, inflexibility, overhead) and outlines the goals for Slack's system: usefulness beyond backend services, ease of use for simple cases, power for complex analysis, real-time triage capabilities, and queryable raw span data. The post contrasts the traditional trace model with the proposed causal graph model, emphasizing the benefits for ad hoc analysis and broader applicability.
  5. Into the CloudsThis post details the technical implementation of extending Slack's push notification system to support all-purpose notifications beyond direct messages and channel posts. It covers modifications to the Android notification payload model to include `teamId`, `userId`, and `subtype`, and the creation of a 'Trampoline' activity to handle deep link intents. The post also addresses backward compatibility challenges, such as handling unknown notification types by swallowing them, and outlines the logic for managing notification display based on type (mention vs. info) and existing notifications. On iOS, it describes the deep link handling mechanism using a `DeepLinkHandler` and `DeepLinkRouterProtocol` to manage routing based on URL structure and app state. Finally, it details the analytics tracking mechanism for notification events and the initial success metrics from a phased rollout of a new notification for team member invites.