Blogs›Shopify›Go Worker Pools for Event Processing
This feature thread tracks the development and enhancement of Shopify's event processing capabilities, focusing on scaling server-side data sharing. Initial efforts involved addressing consumption lag in Kafka by processing events in batches. This post details the implementation of the Go worker pool pattern to control the concurrency of goroutines, significantly increasing event processing performance from 7.75 thousand events per second per pod to over 21 thousand events per second per pod, and ultimately handling up to 46 thousand events per second during peak loads like BFCM.
9/9/2022
Introduced the Go worker pool pattern to the Server Pixels service to manage event processing concurrency. Implemented a fixed number of workers that pull jobs from a Go channel, processing event batches in a controlled manner. Utilized WaitGroups to ensure all in-flight tasks are completed before shutdown. This resulted in a 170% increase in event processing throughput per pod.