Introducing the Together AI Batch API: Process Thousands of LLM Requests at 50% Lower Cost

Introduces a new Batch API for LLM inference, enabling asynchronous processing of large volumes of requests (up to 50,000 per batch, 100MB file size) with a best-effort 24-hour completion window. This offers a 50% cost saving compared to real-time inference. The API supports dedicated rate limits (10 million tokens enqueued per model) and provides real-time progress tracking. It includes a Python SDK example for uploading files, creating batch jobs, monitoring status, and downloading results. Best practices for optimal batch sizes, model selection, and error resilience are also provided.