Dollars per token considered harmful

This post argues for a shift in thinking from 'dollars per token' to 'dollars per request' when self-hosting LLM inference. It explains how API providers use tokens for pricing but application developers should focus on requests, as that's what users care about and what scales costs. The post details how this request-centric view impacts latency estimation, replica scaling, and cost calculation, advocating for Modal's platform as a solution for achieving lower dollars per request.