Scaling Personalized Marketing for Multi-Tenant Commerce Platforms

This post details the architectural evolution of Instacart's multi-tenant marketing platform, focusing on scaling personalized marketing for white-label Storefront Pro retailers. Key technical contributions include: 1. Designing a system to manage hundreds of isolated retailer workspaces within a third-party vendor, ensuring data isolation and brand integrity. 2. Building Instacart-specific tooling for retailers to configure campaigns, customize templates, and target audiences, integrated into the existing retailer platform. 3. Reusing the existing Campaigns Engine for audience evaluation and personalization, emitting customer-level events to a streaming platform. 4. Implementing a stream consumer and batching layer to group customer events into batches of up to 50, significantly improving throughput and reducing pressure on downstream APIs, aligning with third-party provider rate limits. 5. Developing a horizontally scalable CRM Service with asynchronous Sidekiq workers for reliable message preparation and delivery, including idempotency validation and durable result persistence. 6. Addressing operational challenges like domain configuration for sender addresses using an Instacart-agnostic domain and automating IP warming for new retailers to maintain sender reputation.