AI Model Co-Design: Hardware-Friendly LLM Design | NVIDIA Technical Blog

This post details how to design LLMs with hardware-friendly linear layers to optimize inference for throughput and interactivity. It emphasizes aligning linear layer dimensions with GPU tile sizes (multiples of 128, ideally 256 or 512) and a width-over-depth aspect ratio to maximize arithmetic intensity and GPU utilization. It also introduces NVFP4 quantization for high throughput and minimal accuracy loss, and discusses expert parallelism (EP) and hybrid parallel strategies in TensorRT-LLM for scaling MoE models across multi-node Blackwell NVLink systems.