Facebook Channel Product Sync Scalability

Shopify has evolved its Facebook channel integration by scaling product synchronization capabilities. Initial efforts involved establishing the Facebook channel and growing its merchant base. Recent optimizations have addressed the serial nature of the product synchronization backend by introducing concurrent processing of product updates per shop and per product. This was achieved by leveraging Sidekiq Pro's batching functionality, where a supervisor job enqueues a batch of product sync jobs that run concurrently. The supervisor waits for all product sync jobs to complete, holding a lock on the shop's domain to prevent race conditions. After individual product sync jobs complete, the supervisor re-enqueues itself if more products need syncing, increasing worker utility. This has resulted in a significant decrease in the number of products needing sync and has made the syncing horizontally scalable.