TechPrufer
Sign inGet started

Blogs›Together AI›Batch API for LLM Inference

Batch API for LLM Inference

Batch API for LLM Inference

1
posts
2025

Together AI introduces a Batch API for processing large volumes of LLM requests asynchronously. This feature allows users to submit up to 50,000 requests in a single batch file (up to 100MB) for non-urgent workloads, offering a 50% cost reduction compared to real-time inference. The API supports progress tracking through various stages (validation, queuing, processing, aggregation) and provides results within a best-effort 24-hour window. It integrates with the Files API for uploads and job creation, and the `together` Python client for programmatic interaction. The Batch API has dedicated rate limits independent of real-time usage, including a maximum of 10 million tokens enqueued per model.

2025

Introducing the Together AI Batch API: Process Thousands of LLM Requests at 50% Lower Cost

6/11/2025

This post introduces the Together AI Batch API, a new system capability for processing LLM requests in batches. It details the workflow for preparing, uploading, submitting, and monitoring batch jobs, including sample Python code and input format. Key technical aspects highlighted are the 50,000 requests per batch limit, 100MB file size limit, independent rate limits, and a best-effort 24-hour completion window. The post also outlines best practices for batch sizes, model selection, and error handling.