Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)- Google Developers Blog

This post details the systems engineering playbook for optimizing the Qwen 3.5-397B MoE model on Ironwood (TPU v7x). It introduces a modular, model-agnostic optimization strategy using pre-optimized building blocks and hardware-aware cost models. Key technical contributions include: 1. A hybrid sharding scheme combining 8-way Attention Batch Sharding (DP=8) with 8-way Expert Parallelism (EP=8) for MoE layers, overcoming limitations of traditional Tensor Parallelism for GQA and MoE architectures. 2. Detailed analysis of distributed collective sequences, comparing All-to-All shuffling with the implemented DP/EP hybrid. 3. Performance improvements of approximately 3.1x for decode-heavy and 4.7x for prefill-heavy workloads achieved through optimizations like Gated DeltaNet (GDN) linear attention and Attention Data Parallelism. 4. Integration of these optimizations into open-source serving frameworks like vLLM and SGLang.