Comparing tokens per second across LLMs

This post details how to accurately compare tokens per second (TPS) across different LLMs by accounting for tokenizer efficiency variations. It explains that different LLMs use different tokenizers (e.g., Llama 3 with tiktoken, Llama 2 with LlamaTokenizer, Mistral models with their own tokenizers) and that these tokenizers vary in efficiency depending on the input text (e.g., code vs. prose). The post provides a method for adjusting TPS metrics to reflect real-world use by curating representative inputs, running them through the tokenizers of old and new models, calculating relative efficiency, and adjusting TPS calculations accordingly. It also mentions hardware and software optimizations like H100 MIG GPUs, TensorRT-LLM, and FP8 precision as relevant for achieving performance targets.