TechPrufer
Get started
Cerebras
Cerebras logoCerebras · Feature trail

Disaggregated AI Inference

37 posts · Mar 2025 – Aug 2026

What this trail covers

This post details advancements in disaggregated AI inference, splitting prefill and decode onto different machines, potentially using heterogeneous hardware. This architecture aims to significantly improve throughput per kilowatt and reduce latency, especially tail latency, by optimizing hardware for each phase of LLM inference. The partnership with AWS, utilizing Trainium for prefill and Cerebras CS-3 for decode, is highlighted as a key enabler of this new paradigm, offering substantial improvements. This post further details hardware-aware training and fine-tuning techniques (Microscaling Mx formats, speculative sampling, unstructured sparsity) to optimize LLM inference on Qualcomm Cloud AI 100, aiming for up to 10x acceleration by reducing memory footprint and compute requirements without impacting accuracy.

All Cerebras trails
  1. Introducing Cerebras CS-4: The Fastest AI Gets FasterIntroduces the Cerebras CS-4 system with Wafer Scale Engine 3 Turbo processors, achieving up to 30x faster inference and 10x more throughput per watt than CS-3. Details the Nexus rack-scale platform architecture, including pluggable compute backpacks, high-density power delivery, and a new modular I/O subsystem. Highlights native support for disaggregated inference, enabling efficient prefill on other platforms and ultrafast decode on CS-4. Achieves over 1,000 tokens per second on 10T+ parameter models via wafer-to-wafer interconnects with latency as low as 2 microseconds.
  2. CerebrasIntroduces a practical implementation of Mixture-of-Experts (MoE) models using a modified nanogpt codebase. Details the architecture of MoE, including expert networks and mixing logic. Explains the implementation of learned routing with auxiliary loss for load balancing. Identifies and diagnoses a critical bug in learned routing with top_k=1 where the router receives zero gradient from the cross-entropy loss, leading to ineffective expert selection. Proposes and implements a 'null expert' solution to boost gradient flow and improve router performance.
  3. How Faster AI Inference Strengthens CybersecurityThis post extends the concept of disaggregated AI inference by focusing on its application to cybersecurity. It details how faster inference, particularly enabled by Cerebras's architecture, allows for more sophisticated AI-driven security operations and the protection of AI applications. The post introduces a tiered architecture for security analysis (fast initial classification, escalated deeper reasoning) and emphasizes how reduced latency transforms security workflows by enabling more context, validation, and reasoning within operational time constraints. It also highlights specific use cases and partnerships in the cybersecurity domain.
  4. Gemma 4 on Cerebras: Fast Multimodal AIThis post demonstrates the practical application of fast multimodal AI inference using Gemma 4 on Cerebras hardware. It showcases three specific use cases: document analysis (achieving a 17x speedup over GPUs), image search, and a rental car damage scout application that processes video frames. The post details the performance metrics for these applications, highlighting the latency benefits of Cerebras for multimodal tasks. It also provides seven practical tips for developers to build fast multimodal applications, focusing on prompt engineering, model configuration, history management, and tool orchestration. The core contribution is the empirical evidence of low-latency multimodal inference and actionable guidance for its implementation.
  5. Gemma 4 on Cerebras—The Fastest Inference is Now MultimodalIntroduces Gemma 4 31B, a multimodal model, to the Cerebras Inference platform, achieving over 1,800 tokens per second. This represents a significant advancement in multimodal AI inference speed and latency, enabling real-time visual and agentic workflows. The post highlights the model's capabilities in image understanding combined with wafer-scale speed, unlocking new product experiences like screenshot-to-insight, long-context summarization, and screenshot-to-patch generation. It also positions Gemma 4 as a reference medium-size model on Cerebras, comparable in intelligence to Claude Haiku but significantly faster.
  7. Cerebras Brings Kimi K2.6 Inference to EnterprisesThis post details the successful deployment and performance benchmarking of the Kimi K2.6 trillion-parameter model on Cerebras hardware. It highlights achieving 981 output tokens per second, a 6.7x improvement over the next fastest GPU cloud and 23x over the median inference provider. The post explains the technical approach, including distributing weights across multiple wafers, streaming activations, and utilizing on-wafer all-to-all communication with over 200x the bandwidth of NVLink. It also mentions performing computation at 16-bit floating point while storing weights in 4-bit, combined with custom kernels and speculative decoding, to achieve this record-breaking performance for trillion-parameter MoE models.
  8. CerebrasIntroduces Multi-LoRA support for Cerebras Inference, allowing multiple LoRA adapters to be used with a single base model for specialized inference. This enables per-request adapter switching, enhancing the flexibility and efficiency of AI applications like coding assistants by allowing specialization for different languages, tasks, and customers.
  9. CerebrasThis post addresses the challenges of scaling Mixture-of-Experts (MoE) models, particularly memory limitations on GPUs and compute utilization issues on Cerebras WSE. It details the limitations of expert parallelism (EP) on GPUs, including communication overheads and load balancing issues. For the Cerebras WSE, it introduces Batch Tiling on Attention (BTA) as a solution to the compute utilization problem by decoupling batch sizes for attention and expert layers, thereby improving throughput and efficiency as MoE models become sparser.
  10. MoE Fundamentals: Why Sparse Models Are the Future of AIThis post introduces the fundamental concepts of Mixture-of-Experts (MoE) models, explaining their necessity for scaling AI beyond current compute limitations. It details the mechanism of conditional parameter activation and contrasts MoE with dense models, showing how MoE achieves better loss improvement at the same compute cost. The post also outlines the basic implementation of an MoE layer, including the role of the router and expert networks, and references related posts on routing strategies and debugging.
  11. CerebrasThis post details the concept and benefits of disaggregated AI inference, separating the compute-heavy prefill phase from the memory-bandwidth-bound decode phase. It highlights the partnership between AWS (using Trainium for prefill) and Cerebras (using CS-3 for decode) as a concrete implementation. The post emphasizes the significant gains in throughput per kilowatt, reduced latency (especially tail latency), and improved agent responsiveness due to specialized hardware for each phase. It contrasts the Cerebras CS-3's on-chip SRAM and massive memory bandwidth (21 PB/s) with GPU limitations, citing a 1,000–2,000x higher effective memory bandwidth than NVIDIA B200. The post also discusses the competitive implications and the economic shift towards inference spending.
  12. Why the AI Race Shifted to Speed​​​​This post highlights the increasing importance of inference speed in the AI race, driven by its impact on model iteration and software development velocity. It provides examples of major AI labs prioritizing speed, such as Google's Gemini 3 Flash, Anthropic's faster Claude Opus 4.6, and OpenAI's partnership with Cerebras for GPT-5.3-Codex-Spark. The post explains how fast inference enables recursive self-improvement of AI models and accelerates product development cycles, citing Anthropic's pricing strategy and a hypothetical scenario of two companies developing an AI-powered CRM. It connects this trend to the broader digital economy's historical focus on speed and positions high-speed inference as critical infrastructure for the AI era.
  13. CerebrasThis post announces the deployment of Cerebras CS-3 systems in AWS data centers, available via AWS Bedrock. It details a collaboration with AWS to build a new disaggregated inference architecture that pairs AWS Trainium (for prefill) with Cerebras CS-3 (for decode). This architecture aims to deliver 5x more high-speed token capacity by specializing hardware for each inference phase. The post also mentions that both aggregated and disaggregated configurations will be supported, catering to different workload needs.
  14. CerebrasThis post provides a direct performance, price-performance, energy efficiency, and accuracy comparison between the Cerebras CS-3 and Groq's LPU for LLM inference. It highlights that the CS-3 achieves ~6x higher inference speeds, better accuracy (supporting 16-bit precision natively), lower power consumption, and superior price-performance compared to Groq's LPU. The post also emphasizes Cerebras's advantage in supporting both training and inference, and its superior reliability and scalability due to its wafer-scale architecture versus Groq's many-chip approach. It notes that Groq's LPU is optimized for 8-bit models and often requires quantization, leading to potential accuracy trade-offs.
  15. CerebrasThis post provides a direct performance, cost, and power comparison between the Cerebras CS-3 and Nvidia DGX B200 Blackwell GPUs for LLM inference. It quantifies Cerebras's advantage with specific benchmarks (e.g., 21x faster inference for Llama 3 70B, 2,700+ tokens/sec on gpt-oss-120B vs. 900 on B200). The post attributes Cerebras's superior inference performance to its on-chip memory bandwidth, which minimizes data movement for token generation. It also highlights Cerebras's advantages in training speed (10x faster due to simplified scaling) and its ability to scale to trillion-parameter models on a single logical device. While acknowledging Nvidia's ecosystem breadth, the post concludes that Cerebras offers significant advantages for organizations prioritizing low latency, high throughput, and simplified scaling.
  16. CerebrasThe post details the training and inference of Jais-2, a family of Arabic-centric LLMs, on Cerebras wafer-scale clusters. It highlights the use of Cerebras's unified MemoryX and SwarmX fabric for training, which simplifies distributed training by treating the cluster as a single chip with a monolithic memory pool. For inference, it emphasizes the use of on-wafer SRAM for high bandwidth, achieving 2,000 tokens per second for the Jais 2 70B model, significantly outperforming current frontier models. The post also discusses the model development pipeline, including pretraining, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and GRPO, and presents Jais-2 as a blueprint for sovereign AI development.
  17. Faster inference from Cerebras, Beats BlackwellThis post benchmarks Cerebras CS-3 inference performance against NVIDIA Blackwell GB200, showing Cerebras achieving over 3,000 tokens per second for GPT-OSS-120B, a 5x advantage over Blackwell's 650 tokens per second. This performance is attributed to Cerebras's wafer-scale architecture storing the entire model in on-chip memory, eliminating GPU bandwidth constraints. The post also highlights Cerebras's price-performance advantage, delivering significantly higher tokens per dollar.
  18. Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated] - CerebrasThis post provides a deep dive into the Cerebras architecture, focusing on its HW/SW co-design for deep learning. It details the core architecture with its distributed memory (48KB SRAM per core), enabling full memory bandwidth and performance across all BLAS levels, which is critical for unstructured sparsity acceleration. It also explains the fine-grained dataflow scheduling and microthreading that allow for native sparsity acceleration and high utilization by processing only non-zero data and switching between tensor operations. The post highlights how this architecture addresses the 'Grand ML Demand Challenge' by improving core architecture performance, scale-up, and scale-out.
  19. CerebrasThis post details how NinjaTech AI, a multi-agent AI system, leverages Cerebras hardware for significantly faster AI inference. It highlights a 15x speed improvement in query resolution for their "Instant Reasoning SuperAgent," reducing response times from 1-4 minutes to under 20 seconds. This is attributed to Cerebras's custom chips enabling reasoning, logic, math, coding, agentic workflows, and deep research at a fraction of the cost of proprietary models. The post quantifies the impact with increased weekly usage and higher user ratings.
  20. NinjaTech AI and Cerebras launch fastest Deep ResearchAnnounces the launch of Llama 4 on Cerebras, achieving 2,600 tokens per second, which is 38x faster than 4o-mini and 19% higher in model evaluations. Cerebras demonstrates 19x faster output speed compared to the highest recorded GPU solution, achieving total response time under 1 second for 1000 input and 500 output tokens. The Llama 4 Scout model is available via chat and the Cerebras Inference cloud with competitive pricing and superior price-performance.
  21. CerebrasThis post announces the availability and performance of OpenAI's GPT OSS 120B model on the Cerebras Inference Cloud. It provides specific metrics for inference speed (up to 3,000 tokens/sec, 280ms time to first token) and accuracy on reasoning benchmarks, comparing it favorably against GPU-based solutions. The post also highlights the price-performance advantage of using Cerebras for this model.
  22. CerebrasThis post announces the availability of OpenAI's gpt-oss-120B model on Cerebras Cloud, highlighting its performance at 3000 tokens/sec with a 128k context window. It details the model's Mixture-of-Experts architecture and its competitive benchmark results. The post also provides pricing information and encourages developers to try the model.
  23. Introducing Condor Galaxy 1: a 4 exaFLOPS Supercomputer for Generative AI - CerebrasThis post introduces Condor Galaxy 1 (CG-1), a 4 exaFLOPS, 54 million core AI supercomputer built in partnership with G42. CG-1 is the first in a series of nine supercomputers planned, totaling 36 exaFLOPS. It leverages Cerebras Wafer-Scale Cluster technology, connecting up to 192 CS-2 systems as a single logical accelerator, and weight streaming for data parallelism. CG-1 features 82 terabytes of memory, supports up to 100 trillion parameters, and natively handles 50,000-token sequence lengths. The system is available via Cerebras Cloud and G42 Cloud, offering a simplified programming model for large model training.
  24. CerebrasIntroduces the "Cerebras Scaling Law" which connects inference speed directly to AI model intelligence. Explains how Cerebras's on-chip model storage architecture overcomes GPU bandwidth limitations, enabling inference speeds exceeding 2,500 tokens/s. Demonstrates this with Qwen3-32B, achieving single-second reasoning and outperforming GPT-4.1 in intelligence, speed, and cost. Highlights the shift in AI development from scaling parameters to scaling inference-time computation, enabled by hardware speed.
  25. Sei AI - Revolutionizing Financial Services Support with Cerebras-Powered AgentsSei AI leverages Cerebras for fast AI inference in their agents for financial services, achieving a 60% reduction in latency and a 40% reduction in serverless container runtimes. This performance gain enables 100% conversation coverage for compliance reviews, a significant improvement from the previous 5% sampling.
  26. Compressing KV cache memory by half with sparse attentionIntroduces a method to compress KV cache memory by half in LLMs by adapting dense transformer decoders with sparse attention. This involves synthetic long instruction data generation, selective dense layers based on sensitivity analysis, specialized sliding-window attention with global sink tokens, and auxiliary memory tokens. The post releases two models (Llama-3-CBHybridL-8B and Llama-3-CBHybridM-8B) and a synthetic dataset, demonstrating improved performance on long-context benchmarks like LongBench.
  27. Extending LLM context with 99% less training tokens - CerebrasIntroduces a novel, compute-efficient recipe for extending LLM context length by 99% using ~10B training tokens. This involves RAFT-style synthetic data generation with distractor passages, syntactic questions, and translation, alongside train-time token position ID shifting (simplified CREAM) and RoPE base frequency adjustment. A three-stage training pipeline (CPT@32K, SFT@8K, SFT@32K) is detailed.
  28. Unlocking High-Resolution Computer Vision with Wafer-Scale Technology - CerebrasThis post details how Cerebras's wafer-scale technology and Weight Streaming execution mode accelerate Convolutional Neural Networks (CNNs) for high-resolution computer vision tasks, including image segmentation. It highlights the ability to fit large image segmentation models (up to 50 megapixels) on a single CS-2 system, eliminating the need for GPU clusters and their associated orchestration challenges. The post also discusses how the architecture overcomes bandwidth limitations in normalization operations, a common bottleneck in GPU-based training.
  29. Innovations in Virtual Drug Screening - CerebrasThis post details how the Cerebras Wafer-Scale Engine (WSE) architecture, with its 850,000 AI-optimized cores, high-speed memory, and low-latency on-chip fabric, can accelerate virtual drug screening. It explains that the small neural networks commonly used in drug screening, which rely heavily on matrix-matrix multiplication with small weight matrices, are not efficiently utilized by traditional GPUs. The WSE addresses this by mapping multiple copies of these small models onto the chip and distributing computations across processing elements. This enables simultaneous training and inference of multiple models at high throughput, leading to significant inference acceleration. A specific example shows a reduction in screening time from 183 days to 3.5 days for a large compound library.
  30. Announcing the Cerebras Architecture for Extreme-Scale AI - CerebrasIntroduces the Cerebras architecture for extreme-scale AI, featuring disaggregated memory and compute. Key components include MemoryX for off-chip weight storage and streaming, SwarmX for efficient inter-CS-2 system communication enabling near-linear scaling, and fine-grained dataflow scheduling for accelerating sparse neural networks. This architecture allows for models up to 120 trillion parameters and clustering up to 192 CS-2 systems.
  31. Accelerating Large Language Model Training with Variable Sparse Pre-training and Dense Fine-tuning - CerebrasIntroduces the Variable SPDF framework, which enhances the previous SPDF by densifying during pre-training (Stage-2) after an initial sparse pre-training stage (Stage-1). This approach aims to improve task-agnostic representations and offers finer control over pre-training FLOPs. Experiments show Variable SPDF outperforms SPDF for the same effective sparsity and matches dense baseline metrics for a 6.7B parameter model with 64% fewer pre-training FLOPs. The Cerebras Software Platform (CSoft) is highlighted for simplifying the implementation of this sparse training methodology.
  32. Introducing Sparse Llama: 70% Smaller, 3x Faster, Full Accuracy - CerebrasIntroduces a collaboration with Neural Magic to achieve 70% sparsity in LLMs (specifically Llama) using a combination of one-shot pruning, sparse pretraining on SlimPajama, and fine-tuning. Highlights Cerebras CS-3's native support for arbitrary unstructured sparsity and Neural Magic's DeepSparse engine for accelerated CPU inference. Demonstrates full accuracy recovery for downstream tasks and significant parameter reduction.
  33. Cerebras CS-3 vs. Nvidia B200: 2024 AI Accelerators Compared - CerebrasThis post compares the Cerebras CS-3 with the Nvidia B200 for AI accelerator capabilities. It details the CS-3's wafer-scale architecture, 125 petaflops of AI performance, and disaggregated memory architecture (MemoryX) supporting up to 1.2PB, enabling training of trillion-parameter models. It highlights the CS-3's superior interconnect bandwidth (27 PB/s on-wafer fabric) and performance per watt compared to the B200. The post emphasizes the CS-3's readiness and its advantages in memory capacity, simplified programming model, and cost efficiency for large-scale AI training.
  34. Cerebras CS-3: the world’s fastest and most scalable AI accelerator - CerebrasIntroduces the Cerebras CS-3, a third-generation wafer-scale AI accelerator. Key advancements include: 2x performance increase over CS-2 with no increase in power or cost, 10x improvement in cluster scalability (up to 2048 CS-3 systems), and significantly increased external memory options (up to 1,200 TB) enabling training of models up to 24 trillion parameters. The SwarmX interconnect simplifies distributed computing by presenting the cluster as a single chip. Condor Galaxy 3, a 64 CS-3 system supercomputer, is highlighted as an example of exa-scale performance with single-device simplicity. A partnership with Qualcomm is also announced for 10x faster LLM inference.
  35. Cerebras and Qualcomm Unleash ~10X Inference Performance Boost with Hardware-Aware LLM Training - CerebrasThis post introduces hardware-aware training and fine-tuning techniques to optimize LLM inference on Qualcomm Cloud AI 100. It details Microscaling (Mx) formats (MXFP8, MXFP6, MXFP4, MXINT8) for reducing model memory footprint and improving bandwidth-bound decode performance, with nominal accuracy impact. It also describes speculative sampling using a smaller draft LLM to generate and validate tokens, achieving 1.8x speedup on Qualcomm Cloud AI 100 Ultra. Finally, it discusses unstructured sparsity for model compression, enabling performance gains on hardware designed for sparsity like Qualcomm Cloud AI 100 and Cerebras CS-2 for sparse training.
  36. Introducing DocChat: GPT-4 Level Conversational QA Trained In a Few Hours - CerebrasIntroduced Cerebras DocChat, a series of models for document-based conversational question answering (Cerebras Llama3-DocChat and Cerebras Dragon-DocChat). Achieved rapid training times on a single Cerebras System (hours for Llama3-DocChat, minutes for Dragon-DocChat). Leveraged and enhanced Nvidia's ChatQA approach with synthetic data generation and contrastive loss with hard negatives. Addressed limitations in handling unanswerable questions by upsampling relevant samples. Improved arithmetic performance by synthetically generating Chain of Thought variants of datasets and incorporating NuminaMath-CoT. Enhanced entity extraction by mixing in SKGInstruct dataset. Released model weights, training recipes, and datasets open source.
  37. Can Sparsity Make AI Models More Accurate?Introduces Sparse Iso-FLOP Transformation (Sparse-IFT), a technique to increase AI model accuracy by increasing model size and inducing unstructured sparsity while maintaining iso-FLOPs per layer. Demonstrates accuracy gains of up to 3.5% on CV models and 0.4 perplexity improvement on LLMs. Highlights Cerebras's hardware and software co-design for accelerating unstructured sparsity, enabling efficient application of techniques like Sparse-IFT.
  38. UndatedAccelerating GPT-5.6 Sol Ultrafast with OpenAIIntroduces OpenAI's Ultrafast Mode for GPT-5.6 Sol, powered by Cerebras, achieving up to 750 output tokens per second. Highlights the technical advantage of Cerebras's Wafer-Scale Engine architecture in minimizing data movement by keeping model weights on-chip, enabling faster inference for time-sensitive AI applications. Provides benchmark data showing significant speedups compared to other models and modes on tasks like Humanity's Last Exam and GDP-Val.