TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Agentic Inference Scheduling

1 post · Aug 2025 – Jul 2026

What this trail covers

Together AI continues to advance agentic inference scheduling with the introduction of ThunderAgent, a novel program abstraction for agentic LLM request scheduling. This system significantly improves throughput and reduces latency for agentic workloads by treating agentic workflows as schedulable programs. It addresses KV cache thrashing by pausing low-priority workflows under memory pressure and resuming them onto nodes with available capacity, leading to near-linear scaling on multi-node clusters and compatibility with existing inference optimizations. This builds upon previous work in AI agent orchestration and autonomous data scientist agents.

All Together AI trails
  1. ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at ScaleIntroduces ThunderAgent, a system that treats agentic workflows as schedulable programs rather than independent requests. This program-level scheduling mitigates KV cache thrashing by pausing low-priority workflows under memory pressure and resuming them onto nodes with available capacity via a global waiting queue. This approach achieves up to 2.5x higher single-node throughput and 2.4x speedup on an 8-node cluster with near-linear scaling. ThunderAgent is compatible with existing inference optimizations like speculative decoding and KV cache offloading.
  2. How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference SystemsThis post details the application of AI agents to automate complex engineering tasks, specifically focusing on accelerating LLM inference via speculative decoding. It outlines six key patterns for building automation agents: 'Good Tools', 'Documentation', 'Safe Execution' (Infrastructure Patterns), and 'Manage Parallel Sessions', 'Manage Wait Time', 'Progress Monitoring' (Behavioral Patterns). The case study demonstrates how these patterns were applied to an automated speculator training pipeline, reducing manual intervention and turnaround time. The post introduces ThunderAgent as a system that treats agentic workflows as schedulable programs to manage KV cache thrashing and improve throughput.