Mamba-3

What this post added

Introduces Mamba-3, a new state space model (SSM) specifically optimized for inference efficiency. Key technical contributions include: a more expressive recurrence formula derived from an exponential-trapezoidal discretization scheme, complex-valued state tracking for richer dynamics, and a multi-input, multi-output (MIMO) variant that improves accuracy with minimal impact on decode latency. The architecture also incorporates QKNorm (BCNorm) for training stabilization and removes the short convolution by integrating convolution-like mechanisms within the SSM recurrence. Empirical results show Mamba-3 outperforming Mamba-2 and other linear alternatives on language modeling and retrieval tasks, with MIMO further boosting accuracy. The post also highlights the potential for hybrid models combining linear layers with self-attention for future LLM architectures.