2,000 robots walk into a shop: Simulated A/B testing (2026) - Shopify

This post details the engineering efforts behind SimGym, a simulated A/B testing platform for Shopify merchants. It describes the transition from API-based LLM calls to self-hosted open-source models (`gpt-oss-120b`) and the subsequent infrastructure and optimization challenges. Key contributions include: developing custom FlashInfer kernels for long-context speculative decoding on Blackwell GPUs, integrating async scheduling, guided decoding, and speculative decoding for agentic inference, and leveraging NVIDIA's EAGLE-3 speculative decoding head. The post also discusses the use of MIG partitioning for doubling serving instances, leading to significant latency reduction and throughput increase. Prompt optimizations for prefix caching and investigations into reducing reasoning effort are also detailed. The work involved deep collaboration with NVIDIA and CentML to tailor LLM serving for agentic workloads, moving beyond standard chatbot assumptions to handle correlated, long-context requests with JSON schema enforcement.