Inference characteristics of Llama · Cursor

This post provides a detailed mathematical analysis of transformer inference, specifically comparing Llama-2-70B with GPT-3.5-turbo. It derives formulas for FLOPs and memory bandwidth per token for both prompt processing and token generation. The analysis highlights that Llama is more cost-effective for prompt-dominated tasks and less so for completion-heavy workloads, offering specific guidance on when to use Llama versus GPT-3.5 based on workload characteristics and hardware constraints (e.g., 2x A100 GPUs). It also discusses memory requirements for the KV cache and model weights, and briefly touches upon techniques like quantization, Mixture of Experts, and speculative sampling for inference optimization.