Blogs›Shopify›Facebook Channel Product Sync Scalability

Facebook Channel Product Sync Scalability

posts

2017

This feature thread tracks the development and enhancement of the Facebook channel integration, focusing on scaling product synchronization capabilities. Initial efforts involved establishing the Facebook channel and growing its merchant base. Recent optimizations have addressed the serial nature of the product synchronization backend by introducing concurrent processing of product updates per shop and per product. This was achieved by leveraging Sidekiq Pro's batching functionality, where a supervisor job enqueues concurrent product sync jobs and manages their completion. The supervisor also implements a lock on the shop's domain to prevent race conditions at the product record level. Immediate re-enqueueing of supervisor jobs after completion further enhances worker utilization. These changes have significantly improved throughput, enabling horizontal scalability by adding more servers, and have shifted the system from a state of increasing backlog to actively decreasing it.

2017

How We Optimized Product Syncing for Facebook - Shopify

8/14/2017

This post details the optimization of the Facebook channel's product synchronization system to achieve horizontal scalability. The core problem identified was the serial processing of product updates in the backend pipeline. The solution involved replacing the serial product sync path with a concurrent one. This was implemented by using Sidekiq Pro's batching functionality, where a supervisor job enqueues a batch of product sync jobs that run concurrently. The supervisor holds a lock on the shop's domain to prevent race conditions. After product sync jobs complete, the supervisor re-enqueues itself if more products need syncing, improving worker utilization. This resulted in a significant increase in throughput and enabled horizontal scaling by adding more servers.