Rebuilding Notion’s lexical search reindexer

This post details the migration of Notion's lexical search reindexer from a custom ECS-based Node.js system to an Apache Spark-native pipeline. Key technical contributions include: 1. Replacing Snowflake dependency with nightly Spark jobs materializing flat files for metadata. 2. Shifting from direct Elasticsearch writes to generating Elasticsearch native snapshot files using two Spark jobs: a shard partitioner and a snapshot writer. The snapshot writer utilizes embedded Elasticsearch nodes within Spark tasks to produce shard-level snapshots, which are then uploaded to S3. Addressed challenges with executor disk space and EBS throughput by using instance-store NVMe executors and coalescing tasks per shard. 3. Porting document generation logic from TypeScript on Node.js to Scala on Spark, eliminating ECS and its associated operational overhead.