Inference characteristics of Llama · Cursor

What this post added

This post provides a detailed technical breakdown of Llama's inference characteristics. It derives formulas for FLOPs and memory bandwidth per token, distinguishing between prompt processing and token generation. It quantifies the memory requirements for model weights and KV cache, and analyzes the trade-offs between prompt-dominated and completion-heavy workloads when comparing Llama to GPT-3.5. The post also includes a primer on transformer math and discusses factors influencing inference costs.