Building observability into Notion’s dead-letter queue

This post details the integration of enhanced observability into Notion's dead-letter queue (DLQ) system, building upon the existing DLQ Explorer. Key contributions include: 1. Leveraging AWS Athena with partition projection on date and hour columns for efficient querying of S3-stored dead-task records, avoiding new data pipelines. 2. Implementing a robust IAM and KMS setup across multiple regions and environments for Athena access. 3. Designing a query layer with a user-friendly interface that allows filtering by cell, environment, date range, event name, workspace ID, and free text, with partition pruning applied before other filters. 4. Developing a first-class re-enqueue workflow that streams tasks from S3 to an SQS queue, includes confirmation modals with justification, and logs detailed audit trails of who, when, and why tasks were re-enqueued.