How Glean uses Apache Beam Streaming to efficiently read a large corpus from MySQL

This post details the implementation of a streaming pipeline using Apache Beam with the Google Cloud Dataflow runner to efficiently read and process a large corpus from MySQL. It describes the creation of a custom UnboundedSource and UnboundedReader that queries a cache of documents maintained by a scanner object. The scanner issues parallel database reads, fetching documents in batches ordered by document ID to ensure each document is processed at most once per full scan. This approach replaces a batch job, enabling daily reprocessing of documents and improving scalability and cost-effectiveness.