TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Go Worker Pools for Event Processing

1 post · Sep 2022

What this trail covers

Shopify has evolved its event processing capabilities by implementing the Go worker pool pattern to control concurrency and improve performance. This has led to a significant increase in event processing throughput, enabling better preparation for high-traffic events and reuse of the pattern across different pipeline components.

All Shopify trails
  1. Leveraging Go Worker Pools to Scale Server-side Data Sharing - ShopifyThis post details the implementation of the Go worker pool pattern to address scalability issues in the Server Pixels service. The solution involves creating a fixed number of worker goroutines that retrieve jobs from a channel, process them, and send results to another channel. Wait groups are used to ensure all in-flight tasks are completed before shutdown. This resulted in a 170% increase in event processing performance per pod, from 7.75 thousand to 21 thousand events per second.