Lithium: distributed, ephemeral streaming pipelines

This post introduces the Lithium Platform, a new system for building dynamic and ephemeral streaming data pipelines. It details the concept of 'Workplans' which are provisioned and deprovisioned at runtime, built on Kafka and Kafka Streams. The 'Bring your own Host' model allows consumers to integrate an SDK and host processors closer to their data. The architecture is divided into a control plane (managing workplan lifecycle and resource providers) and a data plane (executing data movement). Key features include tenant isolation, pipeline isolation for pause/resume/remediation, and a workplan specification for defining pipeline parameters. This extends the capabilities of traditional ETL systems by enabling on-demand, short-lived data movement.