TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Mamba-3 Inference-Optimized SSM

1 post · Mar 2026

What this trail covers

Mamba-3 is a new state space model (SSM) designed with inference efficiency as the primary goal. It features a more expressive recurrence formula, complex-valued state tracking, and a MIMO (multi-input, multi-output) variant that boosts accuracy without slowing down decoding. Mamba-3 SISO outperforms Mamba-2, Gated DeltaNet, and even Llama-3.2-1B (Transformer) on prefill+decode latency across all sequence lengths at the 1.5B scale. The team also open-sourced the kernels, built using a mix of Triton, TileLang, and CuTe DSL for maximum hardware performance. The architecture includes QKNorm for training stability, removes the short causal convolution by combining simple biases on B and C after BCNorm with the new discretization-based recurrence, and adopts interleaved MLP layers. The optional pre-output projection for Mamba-3 improves length generalization performance on synthetic tasks at the slight cost of in-context real-world retrieval tasks. Hybrid models combining linear layers with self-attention enable better retrieval compared to a vanilla Transformer.

All Together AI trails
  1. Mamba-3This post details the engineering breakthroughs behind Mamba-3, a new state space model (SSM) optimized for inference efficiency. Key improvements include a more expressive recurrence formula derived from an exponential-trapezoidal discretization scheme, complex-valued state tracking, and a multi-input, multi-output (MIMO) variant. These changes allow Mamba-3 to achieve better performance than Mamba-2 and Transformer baselines on prefill+decode latency without increasing inference latency. The architecture also incorporates QKNorm for training stability and removes the short causal convolution by integrating convolution-like mechanisms within the SSM recurrence. The team open-sourced the Triton, TileLang, and CuTe DSL kernels. The post also discusses the trade-offs between training and inference optimization, and the potential for hybrid models combining linear layers with self-attention.