Aurora

What this post added

Introduces Aurora, an open-source, RL-based framework for continuously updating speculative decoding models from live inference traces. This system addresses the issue of stale draft models in production by enabling real-time adaptation to shifting traffic domains and outperforming static speculators. Key technical contributions include a serve-to-train flywheel powered by RL, decoupled inference and training servers, lazy and non-disruptive synchronization, a Tree Attention mechanism for processing speculative decoding results, and adaptation to distribution shift through continuous learning.