Why MoE Models Get More From Speculative Decoding | Cohere

This post validates and extends the prediction of a non-monotonic speedup curve for speculative decoding (SD) with Mixture-of-Experts (MoE) models. It analyzes the impact of temporal correlation in expert routing on SD verification cost, showing that expert overlap between adjacent tokens is significantly higher than predicted by independence baselines. The post also uncovers a distinct mechanism for high speedup at batch size 1, related to fixed-overhead amortization. Concrete design implications for co-optimizing model sparsity and speculative decoding are drawn.