Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72 | NVIDIA Technical Blog

This post details the deployment and performance optimization of the Qwen3.8-2.4T-A95B, a 2.4T-parameter Mixture of Experts (MoE) model, on the NVIDIA GB300 NVL72 platform. It highlights architectural innovations for long-context inference, including a hybrid full/linear attention mechanism and fine-grained MoE, which enable efficient handling of up to one million tokens. The post presents performance metrics (over 4K tokens/sec/GPU, 350 tokens/sec/user) achieved with FP8 precision using TensorRT-LLM and discusses various open-source inference recipes (SGLang, vLLM, NVIDIA Dynamo) and deployment options (NVIDIA NIM). It also mentions post-training capabilities with NVIDIA NeMo AutoModel and LoRA fine-tuning.