How to double tokens per second for Llama 3 with Medusa

Introduces and demonstrates the use of Medusa, a speculative sampling technique that grafts additional decoding heads onto an LLM to generate multiple draft tokens per forward pass. This technique addresses the memory-bound bottleneck in autoregressive LLM inference by leveraging idle compute resources to increase tokens per second. The post details training Medusa heads for Llama 3 8B, validating output quality with MMLU benchmarks, and deploying the optimized model with TensorRT-LLM. Benchmarks show a 94%-122% increase in tokens per second compared to a baseline deployment.