Plumbing At Scale

This post details the development and deployment of an event sourcing platform in Go, built on top of critical Kafka clusters. It introduces a stream processing framework with capabilities like deduplication, filtering, mapping, aggregation, windowing, and joining. The architecture leverages Kubernetes on AWS for deployment infrastructure, utilizing Horizontal Pod Autoscaler and Cluster Autoscaler for elasticity. ScyllaDB is used as an intermediate metastore for pipelines, and results are exposed via the GrabStats service. The post also outlines the anatomy of a stream processing pod, including triggers, runtime, and the pipeline plugin.