The two AI gateway patterns in production inference

This post introduces and details the two dominant AI gateway patterns in production inference: access gateways (application-side, many providers) and serving gateways (customer-facing, own models). It elaborates on the serving gateway's request processing flow, covering identity establishment, serving path selection (mentioning KV cache locality as an optimization), service protection mechanisms (rate limiting, concurrency, overload protection), usage measurement, and auditable result production. It also outlines key evaluation criteria for AI gateways, focusing on customer model representation, tenant isolation, commercial API support, inference awareness, and deployment location tradeoffs.