How input token count impacts the latency of AI chat tools

This post details an empirical study on the correlation between input token count and Time To First Token (TTFT) for LLM calls, specifically using GPT-4 Turbo. It explains the quadratic time complexity of the Transformer's attention mechanism with respect to context window size and presents data showing a linear relationship between input tokens and TTFT, with an approximate increase of 0.24ms per input token for P95 TTFT. The analysis also contrasts this with end-to-end latency improvements and demonstrates how splitting large prompts can significantly reduce perceived latency.