TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Batch API for LLM Inference

1 post · Jun 2025 – Sep 2025

What this trail covers

Together AI's Batch Inference API offers a streamlined UI for job creation and tracking, universal access to all serverless models and private deployments, a 3000x increase in rate limits (from 10M to 30B enqueued tokens per model per user), and typically 50% lower costs compared to real-time APIs. These enhancements make it the most economical and efficient solution for high-throughput, non-real-time workloads like large-scale text analysis, embedding generation, and synthetic data generation.

All Together AI trails
  1. Improved Batch Inference API: Enhanced UI, Expanded Model Support, and 3000× Rate Limit IncreaseThe Batch Inference API has been significantly improved with a streamlined UI for easier job creation and tracking, universal access to all serverless models and private deployments, and a 3000x increase in rate limits (from 10M to 30B enqueued tokens per model per user). These enhancements result in typically 50% lower costs compared to real-time APIs, making it the most economical and efficient solution for high-throughput, non-real-time workloads.
  2. Introducing the Together AI Batch API: Process Thousands of LLM Requests at 50% Lower CostIntroduces a new Batch API for LLM inference, enabling asynchronous processing of large volumes of requests (up to 50,000 per batch, 100MB file size) with a best-effort 24-hour completion window. This offers a 50% cost saving compared to real-time inference. The API supports dedicated rate limits (10 million tokens enqueued per model) and provides real-time progress tracking. It includes a Python SDK example for uploading files, creating batch jobs, monitoring status, and downloading results. Best practices for optimal batch sizes, model selection, and error resilience are also provided.