TechPrufer
Get started
Cursor
Cursor logoCursor · Feature trail

LLM Inference Cost and Performance Analysis

1 post · Jul 2023

What this trail covers

This post analyzes the inference characteristics of Llama, comparing its cost and latency to GPT-3.5. It details the mathematical underpinnings of transformer inference, including FLOPs and memory bandwidth calculations for both prompt processing and token generation. Key findings indicate that Llama is significantly cheaper for prompt-dominated tasks but more expensive for completion-heavy workloads compared to GPT-3.5. The analysis also touches upon memory requirements for KV cache and model weights.

All Cursor trails
  1. Inference characteristics of Llama · CursorThis post provides a detailed mathematical analysis of transformer inference, specifically comparing Llama-2-70B with GPT-3.5-turbo. It derives formulas for FLOPs and memory bandwidth per token for both prompt processing and token generation. The analysis highlights that Llama is more cost-effective for prompt-dominated tasks and less so for completion-heavy workloads, offering specific guidance on when to use Llama versus GPT-3.5 based on workload characteristics and hardware constraints (e.g., 2x A100 GPUs). It also discusses memory requirements for the KV cache and model weights, and briefly touches upon techniques like quantization, Mixture of Experts, and speculative sampling for inference optimization.