OpenAI's Prompt Cache Has a Secret 15 RPS Ceiling

The post identifies and quantifies a secret rate limit of approximately 15 requests per second on OpenAI's prompt cache. It explains that at higher scales, this limit prevents effective cache utilization, requiring developers to implement their own engineering solutions to manage cache hits and misses, thereby achieving the potential cost savings of prompt caching.