TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Real-time Low-Latency Inference Infrastructure

3 posts · Jul 2025 – May 2026

What this trail covers

Together AI has partnered with Cursor to build and deploy real-time, low-latency inference infrastructure on NVIDIA Blackwell GB200 NVL72 and HGX B200. This includes full throughput on ARM hosts, custom kernels for Blackwell Tensor Cores, and efficient parallelism across GPUs. The platform also supports a streamlined weights-to-production cycle with quantization pipelines using NVIDIA TensorRT LLM, enabling rapid iteration and A/B testing of new models. The focus is shifting towards increasing throughput and reducing latency for LLM inference, leveraging bespoke GPU kernels, a proprietary inference engine, advanced speculative decoding methods, and calibrated/quantized model optimization.

All Together AI trails
  1. Foundational research powering efficient inference at scaleThis post details Together AI's ongoing focus on optimizing inference for production AI systems, emphasizing its importance for unit economics and product viability. It highlights the company's compounding stack of research, systems engineering, and hardware expertise. Key contributions include the production deployment of FlashAttention-4, adaptive speculative decoding via the Aurora system (which learns from live inference traces), full-stack hardware optimization for NVIDIA Blackwell, and intelligent scheduling and batching for high-throughput inference. The post also discusses the economic implications of inference optimization, noting significant cost reductions per token and the compounding advantage of efficiency gains for AI-native companies.
  2. Optimizing inference speed and costs: Lessons learned from large-scale deploymentsThis post details lessons learned from large-scale deployments on optimizing inference speed and costs. Key strategies discussed include: maximizing GPU utilization through quantization (FP8/FP4) and distillation (e.g., DeepSeek-R1 variants), reducing network latency with regional inference proxies, minimizing unnecessary compute by optimizing kernel efficiency, MoE layer execution, prefill paths, and batching/scheduling, leveraging decoding optimizations like MTP and speculative decoding (e.g., ATLAS), selecting appropriate hardware (Blackwell, GB200) and parallelism strategies (tensor, expert), and dynamically shifting GPU capacity across endpoints based on real-time demand. It emphasizes that cost efficiency is achieved through optimizing inference execution rather than solely increasing hardware.
  3. Learn how Cursor partnered with Together AI to deliver real-time, low-latency inference at scaleThis post details the engineering collaboration between Together AI and Cursor to deliver real-time, low-latency inference infrastructure on NVIDIA Blackwell GB200 NVL72. Key contributions include porting the inference stack to ARM hosts, developing custom kernels for Blackwell Tensor Cores, designing efficient parallelism meshes for the GB200 NVL72 topology, and implementing a quantization pipeline using NVIDIA TensorRT LLM for rapid model iteration and A/B testing. The focus is now shifting to improving throughput and utilization.
  4. Together AI Delivers Top Speeds for DeepSeek-R1-0528 Inference on NVIDIA BlackwellThis post details the deployment of DeepSeek-R1-0528 on NVIDIA HGX B200, achieving industry-leading inference speeds through a multi-layered optimization approach. Key contributions include: 1) Bespoke GPU kernels leveraging NVIDIA B200's 5th-gen Tensor Cores and Tensor Memory, developed using the ThunderKittens framework. 2) A unified inference engine that captures prefill to verification into CUDA graphs for compute-communication overlap. 3) The Together Turbo Speculator, an advanced speculative decoding method outperforming standard MTP modules by maintaining higher alignment across lookaheads. 4) Lossless quantization techniques (NVFP4, MXFP4/6/8) preserving accuracy even in MoE layers, compressing weights by 3.6x. Dedicated Endpoints (DEs) are highlighted for further optimization, offering up to an 84 tokens/sec speedup by fine-tuning deployment environments for specific workloads.