TechPrufer
Sign inGet started

Blogs›Together AI›Real-time Low-Latency Inference Infrastructure

Real-time Low-Latency Inference Infrastructure

Real-time Low-Latency Inference Infrastructure

3
posts
2026

Together AI has partnered with Cursor to build and deploy real-time, low-latency inference infrastructure on NVIDIA Blackwell GB200 NVL72 and HGX B200. This includes full throughput on ARM hosts, custom kernels for Blackwell Tensor Cores, and efficient parallelism across GPUs. The platform also supports a streamlined weights-to-production cycle with quantization pipelines using NVIDIA TensorRT LLM, enabling rapid iteration and A/B testing of new models. The focus is shifting towards increasing throughput and reducing latency through techniques like quantization, distillation, regional inference proxies, kernel optimization, advanced decoding strategies (MTP, speculative decoding), strategic hardware selection (Blackwell, GB200), appropriate parallelism (tensor, expert), and dynamic GPU capacity shifting across endpoints. This post details lessons learned from large-scale deployments on optimizing inference speed and costs by maximizing GPU utilization, eliminating compute stalls, strategically selecting decoding techniques, and treating performance tuning as an ongoing discipline.

2026

Foundational research powering efficient inference at scale

5/4/2026

This post details Together AI's ongoing focus on optimizing inference for production AI systems, emphasizing its importance for unit economics and product viability. It highlights the company's compounding stack of research, systems engineering, and hardware expertise. Key contributions include the production deployment of FlashAttention-4, adaptive speculative decoding via the Aurora system (which learns from live inference traces), full-stack hardware optimization for NVIDIA Blackwell, and intelligent scheduling and batching for high-throughput inference. The post also discusses the economic implications of inference optimization, noting significant cost reductions per token and the compounding advantage of efficiency gains for AI-native companies.

Optimizing inference speed and costs: Lessons learned from large-scale deployments

1/22/2026

This post details lessons learned from large-scale deployments on optimizing inference speed and costs. Key strategies discussed include: maximizing GPU utilization through quantization (FP8/FP4) and distillation (e.g., DeepSeek-R1 variants), reducing network latency with regional inference proxies, minimizing unnecessary compute by optimizing kernel efficiency, MoE layer execution, prefill paths, and batching/scheduling, leveraging decoding optimizations like MTP and speculative decoding (e.g., ATLAS), selecting appropriate hardware (Blackwell, GB200) and parallelism strategies (tensor, expert), and dynamically shifting GPU capacity across endpoints based on real-time demand. It emphasizes that cost efficiency is achieved through optimizing inference execution rather than solely increasing hardware.

Learn how Cursor partnered with Together AI to deliver real-time, low-latency inference at scale

1/13/2026

This post details the engineering efforts by Together AI in collaboration with Cursor to establish real-time, low-latency inference infrastructure on NVIDIA Blackwell. Key contributions include: - **Hardware Engineering:** Early rollout and reliable delivery of NVIDIA Blackwell GB200 NVL72 and HGX B200, including fast upgrades and replacements. - **System Optimization:** Full throughput achieved on ARM hosts by tuning the inference stack at the kernel and host levels. - **Custom Kernel Development:** Building kernels for Blackwell's new Tensor Core instructions to maximize hardware throughput. - **Parallelism Design:** Designing parallelism meshes for GB200 NVL72 to manage communication and synchronization overhead. - **Quantization Pipeline:** Implementing a quantization pipeline using NVIDIA TensorRT LLM and NVFP4 on Blackwell to balance compression with model quality for real-time serving. - **Production Deployment:** Establishing a repeatable path for moving trained models to production-like endpoints for testing and deployment, including A/B testing and cutovers.