Production-Ready W4A8: vLLM Integration Explained | Cohere

This post details the implementation of production-ready W4A8 GEMM kernels for NVIDIA Hopper GPUs, integrated into vLLM. It introduces a LUT-based solution for efficient INT4 to FP8 dequantization, a custom CUDA kernel for faster pre-processing, and a refined quantization recipe that uses token masking during calibration and compensates for FP8 range limitations to maintain model quality. The post quantifies performance improvements and accuracy preservation compared to naive FP8 casting and W4A16.