Using asynchronous inference in production

This post introduces asynchronous inference as a new capability on the Baseten platform. It details the workflow, contrasting it with real-time inference and highlighting its benefits for handling long-running tasks, traffic spikes, and request prioritization. Key technical aspects include request queuing, priority-based processing, webhook/cloud storage output, user-configurable retry schemas, detailed model logs, in-queue time metrics, request cancellation, backpressure mechanisms, and heartbeat for stuck requests. The implementation aims to abstract complexity for a seamless developer experience.