TechPrufer
Get started
Glean
Glean logoGlean · Feature trail

Apache Beam Streaming for MySQL Corpus Processing

1 post · Aug 2026

What this trail covers

Glean is enhancing its data processing capabilities by implementing Apache Beam streaming pipelines to efficiently read and process a large corpus from MySQL. This approach replaces costly and fragile batch jobs, ensuring each document is re-processed at least once daily. The system utilizes a custom UnboundedSource and UnboundedReader within Apache Beam on Google Cloud Dataflow, with a scanner object managing parallel database reads and tracking cache state for infinite data streaming. This enables scalable and cost-effective processing of hundreds of millions of documents daily.

All Glean trails
  1. How Glean uses Apache Beam Streaming to efficiently read a large corpus from MySQLThis post details the implementation of an Apache Beam streaming pipeline on Google Cloud Dataflow to process a large corpus from MySQL. It introduces a custom UnboundedSource and UnboundedReader, managed by a static scanner object that issues parallel database reads. The scanner tracks cache state to ensure each document is processed at least once daily and to enable infinite data streaming. This approach replaces previous batch jobs, offering improved scalability and cost-effectiveness by leveraging underutilized streaming pipeline resources.
  2. Tripling streaming pipeline throughput by monitoring and debugging MySQL inefficienciesThis post details how Glean optimized MySQL read queries within their Apache Beam streaming pipelines to triple throughput. The inefficiency was identified by monitoring the in-memory buffer being empty and observing that larger offsets in `LIMIT` clauses were slowing down queries. The root cause was MySQL fetching `(n+1)*k` documents and discarding the first `n*k`. The solution involved modifying the query to force late row lookups, retrieving only the required `k` documents. This reduced query latency from ~4 seconds to ~0.1 seconds for a test case, leading to a tripling of the document processing rate.