TechPrufer
Get started
Cerebras
Cerebras logoCerebras · Feature trail

AI Inference Latency Optimization

52 posts · Mar 2025 – Aug 2026

What this trail covers

This post details advancements in AI inference latency optimization by enabling LLMs to efficiently leverage extended contexts through strategic planning and iterative steps to access and integrate relevant information from extended data sources. This approach, ReadAgent, effectively manages and processes information beyond inherent context windows, supporting long-context tasks by decomposing complex tasks and utilizing iterative planning steps. It achieves improved performance on long-context tasks. Cerebras DocChat, a new iteration of models for document-based conversational QA, is released. Cerebras Llama3-DocChat was trained in hours on a single Cerebras System, and Cerebras Dragon-DocChat was fine-tuned in minutes. These models incorporate insights from research on document-based Q&A, synthetic data generation for unanswerable questions, Chain of Thought for arithmetic tasks, and instruction tuning for entity extraction. The models and training recipes are open-sourced.

All Cerebras trails
  1. Cerebras Serves GPT-5.6 Sol at 750 Tokens/SecondThis post details how Cerebras WSE-3 hardware enables GPT-5.6 Sol to achieve up to 750 tokens/second by eliminating the GPU memory wall. The WSE-3 distributes 44 GB of SRAM directly beside its 900,000 cores, providing 21 petabytes per second of aggregate bandwidth. This allows model weights to stay close to compute, with only activations moving between CS-3 systems when a model is partitioned across multiple wafers. This architecture significantly reduces the synchronization overhead compared to conventional GPU clusters, leading to a 5.6x speedup on GDP-Val tasks and a 6.9x speedup on Humanity's Last Exam tasks.
  2. Ultrafast Frontier Inference | Cerebras Hot Chips 2026Introduces the CS-4 system built on the Nexus rack-scale platform, detailing its modular compute backpack architecture with improved power delivery (0.5mm from wafer), integrated water-conditioning for cooling, and centralized front-of-rack power. Previews CS-5 targeting 10,000 output tokens/sec/user for open-source models and 5,000 tokens/sec/user for frontier models, and CS-6 which will integrate wafer-scale SRAM with 3D-stacked DRAM. Highlights the architectural advantage of Cerebras's on-wafer fabric bandwidth (53.5 PB/s) compared to conventional multi-GPU systems, emphasizing reduced latency, power, cost, and failure points.
  3. Introducing Cerebras CS-4: The Fastest AI Gets FasterIntroduces the Cerebras CS-4 system, featuring Wafer Scale Engine 3 Turbo processors, designed for significantly faster AI inference (up to 30x over GPUs) and high token capacity (1000+ tokens/sec for 10T+ parameter models). Highlights include reduced wafer-to-wafer interconnect latency (as low as 2 microseconds), improved throughput per watt (10x over CS-3), native support for disaggregated inference (pairing with prefill engines like AMD Helios and AWS Trainium), and the new Nexus rack-scale platform with a modular design for compute, power, and I/O, enabling faster manufacturing and deployment.
  4. How Faster AI Inference Strengthens CybersecurityThis post details how faster AI inference speeds can provide a competitive advantage in cybersecurity. It discusses two main areas: AI for Security (using AI to improve detection, investigation, etc.) and Security for AI (protecting AI applications themselves). The post highlights a tiered architecture pattern where initial filtering is done by simpler models, with more complex reasoning models used as an escalation point. Faster inference is crucial at both levels to maintain responsiveness and allow for higher-quality thinking in time-sensitive security operations. The Cerebras wafer-scale architecture is presented as a solution to reduce data movement and distributed system overhead, enabling faster inference for cybersecurity applications. Specific examples of partnerships with CrowdStrike, Armis, and Operant AI are provided to illustrate these concepts.
  5. Gemma 4 on Cerebras: Fast Multimodal AIThis post demonstrates the practical application of Gemma 4 31B on Cerebras hardware, achieving ~2,300 tokens/sec for multimodal inference. It details three use cases: document analysis (achieving a 17x speedup over GPU providers), image search, and a rental car damage scout application that processes video frames for damage detection within 5 seconds. The post highlights the importance of designing the entire application for speed, not just the model endpoint, and provides seven tips for building fast multimodal applications, including prompt templating, generation defaults, intentional thinking mode, clean history management, image-before-text ordering, structured long context input, and simple tool loops. The core contribution is showcasing how low-latency multimodal inference enables interactive product experiences and changes the design possibilities for developers.
  7. Gemma 4 on Cerebras—The Fastest Inference is Now MultimodalThis post introduces the integration of Gemma 4, a multimodal model, with Cerebras's inference platform, achieving 1,851 output tokens per second. This represents a significant advancement in multimodal AI inference speed, enabling real-time visual and agentic workflows. The post highlights the performance gains for tasks like screenshot-to-insight and screenshot-to-patch, emphasizing how wafer-scale speed transforms application development beyond mere acceleration.
  8. The Economics of AI ReasoningThis post analyzes the economic trade-offs of enabling 'test-time compute' for reasoning in LLMs. It quantifies the cost increase (6x more tokens, 7-11x longer completion times) versus the performance benefit (10-20% improvement) for reasoning. It categorizes reasoning types (interleaved, adaptive, configurable) and discusses its utility for complex tasks. Crucially, it highlights that for many common agentic tasks, disabling reasoning can lead to significant cost savings (85% cheaper) and improved performance due to reduced compaction, advocating for reasoning as a configurable speed/cost control.
  9. Which is faster: Gemini 3.5 Flash or Kimi K2.6 on CerebrasThis post benchmarks Kimi K2.6 on Cerebras against Gemini 3.5 Flash, showcasing Cerebras's superior inference speed. Key metrics include output tokens per second (981 vs. 181), end-to-end response time (5.6s vs. 17.5s), and time-to-first-token for voice agents (452ms vs. 960ms). The advantage is attributed to Cerebras Wafer Scale Engines' on-chip storage, which avoids external memory loading.
  10. Cerebras Brings Kimi K2.6 Inference to EnterprisesCerebras has achieved 981 output tokens per second running the Kimi K2.6 trillion-parameter open-weight model, a world record for this class of model. This performance is attributed to the Cerebras Wafer-Scale Engine's architecture, which distributes 4-bit weights across multiple wafers and streams activations between them. On-wafer network fabric provides over 200x the bandwidth of NVLink for all-to-all communications between layers. Custom kernels and speculative decoding further contribute to the high throughput. This enables agentic coding workflows to achieve near-instantaneous response times, with a 10,000-token input request (including prompt processing, reasoning, and generating 500 output tokens) completed in 5.6 seconds, a 29x improvement over the official Kimi endpoint.
  11. CerebrasIntroduces Multi-LoRA (multi-adapter support for Low-Rank Adaptation) on Cerebras Inference. This feature allows teams to apply a different LoRA adapter per request to a single shared base model, enabling specialization for various domains, tasks, customers, and workflows. It supports deploying LoRAs in HF PEFT format and switching adapters on a per-request basis. An example use case is provided for coding assistants to specialize by language, task, and customer, and for customer-facing assistants to personalize for individual customer environments.
  12. CerebrasThis post details Cognition's co-design of their SWE-1.6 and SWE-grep agents with Cerebras Inference. It highlights the achievement of ~5x faster coding performance (up to ~950 tokens/second) compared to GPUs, enabling developers to explore large repositories and make precise code changes in under five seconds. The post emphasizes improvements in model UX, including more parallel tool calls, reduced looping, and greater reliance on agent tools over terminal commands. SWE-grep's Fast Context sub-agent collapses context gathering from tens of seconds into seconds. The combined optimization of model, agent harness, and inference layer delivers a cohesive agent experience tuned on real engineering workflows and model UX.
  13. CerebrasThis post highlights the increasing mainstream adoption of fast AI inference, driven by its necessity for responsive user experiences, efficient agent reasoning, and new application classes. It emphasizes Cerebras's architectural advantage in delivering ultra-low-latency inference through its wafer-scale engine, resulting in up to 15x faster performance than conventional GPU systems. The core contribution is the expansion of the Cerebras ecosystem to make this speed broadly accessible. This includes increased data-center capacity, expanded cloud availability, and deep integrations with over 20 categories of developer tools and platforms (e.g., agentic frameworks, LLM frameworks, observability tools). The post also details support for a wide range of open models, optimized for low latency, and offers flexibility for on-premises and private deployments. The focus is on reducing friction for adoption through self-serve cloud experiences and enterprise-ready procurement via cloud marketplaces.
  14. CerebrasThis post discusses how Cerebras' Wafer-Scale Engine, with its on-chip memory and high token-per-second throughput, can significantly reduce the latency overhead associated with MCP protocols in agentic AI workflows. It posits that faster inference infrastructure makes the structured, auditable nature of MCP more practical for enterprise deployments by mitigating the performance cost of tool calls.
  15. Why the AI Race Shifted to Speed​​​​This post argues that the AI race has shifted from model intelligence to inference speed. It highlights examples of faster models from Google (Gemini 3 Flash), Anthropic (Claude Opus 4.6 Fast), and OpenAI (GPT-5.3-Codex-Spark). The post explains that inference speed is crucial because it directly impacts the rate of model iteration for AI labs and software development for the broader economy. It cites OpenAI and Anthropic's use of their own coding models for self-development and notes that Anthropic prices faster models at a significant premium. The post further illustrates the business impact of fast inference through a hypothetical AI-powered CRM development scenario and references Stripe's annual letter on rapid ARR growth, attributing it to agentic coding. It concludes that high-speed inference is becoming critical infrastructure, akin to fast computers and internet connections in previous eras, and that Cerebras's focus on speed positions it well for this trend.
  16. How to stop your autoresearch loop from cheatingThis post details experiments using Andrej Karpathy's autoresearch framework to autonomously optimize AI models. It describes the implementation of a bash harness for Codex to enable looping and A/B testing in training optimization, and the application of autoresearch to dynamic expert swapping for inference optimization on consumer GPUs. The post analyzes the results, emphasizing the convergence of different agents on similar optimizations, the impact of proposal quality on cost, and the critical role of environment design and task framing in productive exploration. It also identifies infrastructure gaps in sandboxing and tooling that hinder autonomous research.
  17. CerebrasIntroduces a disaggregated inference architecture that separates prefill (handled by AWS Trainium) and decode (handled by Cerebras CS-3) phases of AI computation. This architecture utilizes high-speed interconnects (Amazon's EFA) to improve inference speed and token output capacity by leveraging the specialized strengths of each hardware component.
  18. CerebrasIntroduces Codex Spark, a low-latency AI coding model (1200+ tokens/sec) running on Cerebras WSE hardware, enabling real-time developer-AI collaboration. Provides practical strategies for leveraging Spark's speed, including treating it as a pair programmer, using rapid validation cycles (pre-commit hooks, test suites, linting, browser QA), exploring multiple implementation paths, managing context windows with fresh sessions, externalizing state via Git and markdown files (PLAN.md, PROGRESS.md, VERIFY.md), and applying tighter permissions and restrictions (e.g., max diff size, file modification rules). Also suggests a workflow of using a slower Codex model for planning and Spark for rapid execution.
  19. CerebrasThis post provides a comparative analysis of Cerebras CS-3 and Groq LPU performance for LLM inference. It highlights Cerebras's advantages in inference speed (~6x higher), end-to-end response time (up to 5x faster), price-performance (up to 6x advantage), energy efficiency (~3x higher compute-per-watt than 8-GPU DGX), and accuracy (native 16-bit precision support). It also notes Cerebras's ability to support both training and inference, unlike Groq's inference-only LPU, and discusses reliability and scaling advantages due to its wafer-scale architecture versus Groq's many-chip approach. Availability is also cited as an advantage for Cerebras.
  20. CerebrasThis post benchmarks the Cerebras CS-3 against Nvidia's DGX B200 Blackwell GPU, demonstrating a 21x faster inference performance for LLMs like Llama 3 70B. It highlights Cerebras's advantage in memory bandwidth, keeping model weights on-chip to reduce latency. The post also details significant price-performance (32% lower cost) and energy efficiency advantages, along with improved accuracy in reasoning tasks. For training, Cerebras offers up to 10x faster time-to-train due to simplified scaling. Availability is broad for CS-3, while B200 is supply-constrained. Reliability and ease of use are rated as even, with Nvidia leading in ecosystem breadth.
  21. CerebrasThis post elaborates on the Cerebras Scaling Law, emphasizing how inference speed is no longer just a usability threshold but a crucial lever for achieving higher AI accuracy. It details how the Cerebras wafer-scale architecture, with its tightly integrated compute and memory (SRAM) and massive on-chip bandwidth, overcomes the memory-bandwidth bottleneck inherent in GPU-based inference. The post contrasts Cerebras's approach with traditional GPU architectures, highlighting its ability to deliver up to 15x faster inference speeds. It presents a trade-off curve between speed and accuracy, suggesting that applications can leverage this speed headroom to achieve greater accuracy within latency budgets. Real-world examples from Tavus, OpenAI Codex Spark, and AlphaSense are provided to demonstrate the practical application of this speed advantage for minimizing latency, accelerating code iteration, and expanding reasoning capabilities.
  22. CerebrasIntroduces OpenAI's GPT-5.3-Codex-Spark model, powered by Cerebras, for real-time software development. Achieves over 1,000 tokens/s for near-instant feedback in live coding environments. Codex-Spark is optimized for fast inference and excels at precise edits, revising plans, and answering contextual questions about codebases, enabling faster iteration cycles for UI visualization, styling refinement, and interface testing. Highlights the Cerebras Wafer-Scale Engine's capability for high-speed inference at thousands of tokens per second per user, with scalability to multi-terabyte memory for trillion-parameter models.
  23. CerebrasThis post details StackAI's integration of Cerebras for ultra-low-latency inference for enterprise AI agents. It highlights how Cerebras enables sub-second latency for complex, multi-step agentic workflows involving RAG, tool use, planning, and verification, which is crucial for regulated industries. The integration results in significant reductions in latency, improvements in throughput stability, and lower workflow-level inference costs. Cerebras' support for private and hybrid deployments is also noted as essential for StackAI's enterprise customers.
  24. CerebrasThis post introduces the concept of 'latency debt' in AI, defining it as the accumulated cost of optimizing models faster than infrastructure, leading to slow user experiences. It quantifies the impact of larger models, reasoning tokens, longer context windows, and increased output lengths on this debt. The post argues that current GPU architectures are becoming a bottleneck for these latency-sensitive AI workloads and proposes Cerebras's Wafer-Scale Engine (WSE) as a superior alternative due to its on-chip data processing, which eliminates memory bandwidth constraints. It also cites major AI companies' investments in alternative inference hardware as validation of this trend. The post concludes by emphasizing the immediate benefits of optimized hardware for AI inference, transforming AI from a bottleneck to a collaborator.
  25. CerebrasThis post introduces GLM 4.7, an open-source model that achieves up to 1500+ tokens per second on Cerebras hardware, making it 20x faster than closed-source competitors. It details GLM 4.7's capabilities in coding, tool use, and reasoning, highlighting its performance on benchmarks like 𝜏²-Bench Telecom, GPQA Diamond, and LiveCodeBench. The post provides 10 rules for optimizing GLM 4.7, focusing on prompt engineering (front-loading instructions, direct language), task management (breaking down tasks, disabling/enabling reasoning), and agentic patterns (using critics, pairing with frontier models). It also introduces the `disable_reasoning` and `clear_thinking` parameters for controlling model behavior on the Cerebras API.
  26. CerebrasThis post introduces GLM-4.7, a new GLM family model from Z.ai, available on Cerebras Inference Cloud. It highlights GLM-4.7's advancements in frontier intelligence for coding, tool-driven agents, and multi-turn reasoning, outperforming other open-weight models on developer benchmarks. Key technical improvements include interleaved thinking and preserved thinking for enhanced reasoning. The post emphasizes GLM-4.7's record speed on Cerebras hardware, achieving ~1,000 tokens per second, and its superior price-performance compared to leading closed models. It also provides migration guidance for users of GLM-4.6.
  27. Fast Inference Finds its GrooveThis post highlights the industry-wide recognition of inference speed as the primary constraint in AI systems, moving beyond the focus on model size. It details how Cerebras's wafer-scale architecture has become the leading solution for large-model inference, citing numerous customer adoptions and performance benchmarks that demonstrate significant speedups (e.g., 57x faster than GPUs for DeepSeek R1 Llama 70B, 10x faster for Qwen3 235B Instruct, 2000 tokens/sec for Qwen3 Coder 480B, 3000 tokens/sec for OpenAI's gpt-oss-120B). It also introduces the "Cerebras Scaling Law," which posits that faster inference leads to smarter AI, and showcases new products and applications enabled by this speed, such as Cognition's SWE-1.5 and NinjaTech AI's Fast Deep Coder. The post also mentions infrastructure expansion and funding rounds as supporting growth.
  28. CerebrasIntroduces the Implicit Chain Transformer (ICT) architecture to address the statelessness bottleneck in standard Transformers for state-intensive tasks. ICT propagates a learnable 'intent' latent vector across time steps, enabling explicit state updates and contextualization. Two propagation strategies are explored: Autoregressive Intent Propagation (Dense) and Periodic Intent Propagation (Sparse). A novel training method, 'Iterative Latent Feedback,' is proposed to maintain parallel training efficiency by using multi-pass approximations. Early evaluations on modulo arithmetic and graph traversal tasks demonstrate ICT's superior accuracy compared to baseline Transformers, with the Dense variant showing better stability on graph traversal due to challenges in compressing block representations in the Sparse variant with shallow networks. Future work includes inference optimization, scaling the Sparse formulation, and investigating training dynamics.
  29. CerebrasThe post details the training and deployment of Jais 2, a family of Arabic-centric LLMs, on Cerebras wafer-scale clusters. It highlights the achievement of 2,000 tokens per second inference speed for the Jais 2 70B model, attributed to Cerebras's on-wafer SRAM architecture and speculative decoding stack. The training process utilized a unified MemoryX and SwarmX fabric, enabling weight streaming and eliminating the need for complex parallelism strategies, leading to near-linear performance scaling. The post also discusses the model development pipeline, including pretraining on 2.6 trillion tokens, supervised fine-tuning, and Direct Preference Optimization (DPO) for alignment.
  30. CerebrasThis post analyzes three fundamental MoE routing strategies: Hash Routing, Learned Routing, and Sinkhorn Routing. It details their mechanisms, performance tradeoffs (expert utilization vs. specialization), and practical implementation challenges. Learned Routing is highlighted as the industry standard due to its performance, while acknowledging its router collapse issue. Sinkhorn Routing is presented as a method to achieve per-layer load balancing but with scaling and gradient detachment complexities. The post also discusses common engineering tricks layered on these core methods, such as capacity factors and auxiliary loss, and references specific models like DeepSeek-V3 and Qwen2 for their routing implementations. It concludes by emphasizing the fragility of MoE training and the importance of understanding these core routing mechanisms for debugging.
  31. CerebrasThis post details Rox's implementation of dynamic model routing to optimize AI inference latency for agentic workflows. It highlights the use of Cerebras Inference for the 'fast path' which prioritizes low latency and high tokens/sec for real-time user interactions like chat and voice. The post also discusses how Cerebras is used for short, frequent model calls within orchestration loops to maintain flow and reduce overall user-perceived wait times. The availability of Cerebras Inference through AWS Marketplace is also noted as a key enabler for scalable adoption.
  32. Faster inference from Cerebras, Beats BlackwellThis post benchmarks Cerebras hardware against NVIDIA Blackwell GPUs for AI inference, specifically for the GPT-OSS-120B model. It demonstrates Cerebras's continued leadership in inference speed, achieving over 3,000 tokens/sec compared to Blackwell's 650 tokens/sec, a 5x advantage. The post attributes this to Cerebras's wafer-scale architecture storing the entire model on-chip, eliminating GPU bandwidth limitations. It also analyzes price-performance, showing Cerebras offers a better ratio despite a higher per-token cost.
  33. CerebrasThis post demonstrates the practical application of Cerebras Inference in building real-time Reinforcement Learning (RL) style feedback loops for AI development. It details how to integrate Meta's Prompt-Ops and Synthetic-Data-Kit with Cerebras Inference to accelerate prompt engineering and synthetic data generation/curation. Specific technical contributions include: configuring Prompt-Ops for Cerebras Inference with Llama 3.3 70B, showcasing Prompt Duel Optimizer and MIPRO strategies; detailing the Synthetic-Data-Kit pipeline (QA Creation, LLM Judge calls) and its transformation with fast inference for distilling reasoning to an edge model; and emphasizing the iterative nature of prompt strategy refinement enabled by Cerebras's speed.
  34. CerebrasIntroduced REAP (Router-weighted Expert Activation Pruning), a one-shot pruning method for MoE models. REAP measures expert importance by combining router gate-values and expert output magnitude to identify and remove low-impact experts. Demonstrated that pruning is superior to merging for generative tasks due to the 'functional subspace collapse' issue caused by merging. Showcased REAP's ability to prune up to 50% of experts in trillion-parameter models while maintaining high model quality across various generative benchmarks.
  35. CerebrasThis post provides a detailed breakdown of Mixture-of-Experts (MoE) inference arithmetic, clarifying the meaning of terms like '8x7B' and explaining the actual active parameters. It offers in-depth calculations for memory requirements (model weights and KV-cache) and FLOPs (prefill and decode stages) for MoE models. The post contrasts deployment bottlenecks across different hardware setups: single-GPU (memory-bound), multi-GPU (communication-bound), and specialized hardware like Cerebras WSE (compute-bound). It introduces the concept of KV-caching to make decode stage compute linear with sequence length and explains the trade-offs involved.
  36. Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated] - CerebrasThis post provides a deep dive into the Cerebras architecture, focusing on its HW/SW co-design for deep learning. It details the core architecture with its distributed memory (48KB SRAM per core with high bandwidth), fine-grained dataflow scheduling, and hardware support for tensor instructions. It explains how this enables full performance across all BLAS levels and native acceleration of unstructured sparsity. The post also touches upon scale-up and scale-out strategies to address the Grand ML Demand Challenge.
  37. NinjaTech AI and Cerebras launch fastest Deep ResearchIntroduces Llama 4 Scout running at 2,600 tokens/sec, a 38x speedup over 4o-mini, with 19% higher model eval scores. Achieves total response time under 1 second for 1000 input/500 output tokens. Claims 19x faster output speed than highest recorded GPU solution and superior price-performance.
  38. CerebrasThis post announces the availability of OpenAI's GPT OSS 120B model on the Cerebras Inference Cloud, highlighting record-breaking inference speeds (up to 3,000 tokens/sec, 15x faster than leading GPUs) and low latency (280ms time to first token). It emphasizes the model's accuracy on reasoning benchmarks (AIME 2025) and its suitability for agentic and coding applications due to its speed. The post also details a significant price-performance advantage (8.4x tokens/sec/dollar) compared to GPU clouds.
  39. CerebrasAnnounces the availability of OpenAI's gpt-oss-120B on Cerebras Cloud, achieving 3000 tokens/sec at full 128k context. Highlights the model's Mixture-of-Experts architecture, 128k context window, and suitability for agentic workflows. Provides pricing details for input and output tokens.
  40. Qwen3 Coder 480B is Live on CerebrasThis post announces the availability of Alibaba's Qwen3 Coder 480B Instruct model on Cerebras, achieving an unprecedented 2,000 tokens per second. This significantly reduces coding problem resolution time compared to other leading models like Claude 4 Sonnet and Gemini 2.5. The post highlights the model's performance in generating code, enabling developers to generate 1,000 lines of JavaScript in 4 seconds. It also mentions integration with the Cline coding agent for VS Code and availability on Cerebras Inference Cloud, OpenRouter, and HuggingFace, with new subscription plans (Cerebras Code Pro and Max) for accessibility.
  41. CerebrasThis post introduces the application of online Reinforcement Learning (RL), specifically GRPO, to enhance LLM reasoning and planning capabilities. It demonstrates this by teaching LLMs to solve Sudoku puzzles, exploring various puzzle representations (Markdown, Row, Grid) and guidance levels. The work details training methodologies, including dynamic sampling and priority-based sampling for curriculum learning, and analyzes the impact on metrics like avg@4 and pass@4. It also investigates the generalization of trained models to different representations and public benchmarks (AIME, GPQA), showing improved performance in complex task handling.
  42. CerebrasThis post introduces the 'Cerebras scaling law,' which states that faster inference speed directly translates to higher model intelligence. It contrasts this with traditional scaling laws focused on model size and training data. The post highlights how Cerebras's wafer-scale architecture, by storing entire models on-chip, eliminates memory bandwidth bottlenecks and achieves inference speeds over 2,500 tokens/s. This speed enables the use of reasoning models (e.g., Qwen3-32B) in real-time applications, outperforming non-reasoning models and even closed-source frontier models in terms of intelligence and cost-effectiveness.
  43. Reasoning in one second – Try Qwen3-32B on CerebrasThis post introduces the availability of the Qwen3-32B model on Cerebras hardware, achieving 2,400 tokens/s for inference. It highlights the model's performance in code generation, tool-calling, and advanced reasoning, comparing it favorably to GPT-4.1 and Claude Sonnet 3.7. The post quantifies the latency reduction (1.2 seconds for the first token) and cost savings ($0.80 per million output tokens vs. $8 for GPT-4.1), emphasizing its suitability for real-time applications like agents and multi-step reasoning.
  44. Humanizing Digital Technology with NorbyThis post details how Norby leverages Cerebras for AI inference to achieve significant reductions in conversational response latency (87%), improve language switching (74%), and speed up conversation summary processing (88%). It highlights the use of Cerebras' high token processing rates for incremental and parallelized reasoning during conversational pauses, enabling dynamic integration of reasoning throughout the conversational flow. Additionally, it mentions the use of Cerebras' open-source models for advanced turn detection and conversational cue analysis, allowing for rapid and accurate inference of user intentions within approximately 300 milliseconds.
  45. CePO Update: Turbocharging Reasoning Models’ capability using test-time planningThis post introduces an update to the Cerebras Planning and Optimization (CePO) framework, enhancing its ability to turbocharge the capabilities of open-source reasoning models like DeepSeek R1 and Qwen QwQ 32B. The update focuses on test-time planning and verification to achieve unprecedented accuracy, particularly under context length limitations. It demonstrates significant accuracy boosts (20+ points in some instances) and a reduction in unanswered rates (10+ percentage points) for models like AIME and GPQA when context length is restricted to 16k. The enhancements allow reasoning models to fully leverage available context, achieving performance on par with or surpassing Deepseek-R1 on LiveCodeBench and AIME.
  46. Sei AI - Revolutionizing Financial Services Support with Cerebras-Powered AgentsSei AI leverages Cerebras for its financial services support agents, achieving a 60% reduction in latency and a 40% reduction in serverless container runtimes. This performance improvement enables 100% conversation coverage for compliance review, a significant shift from previous sampling-based methods. The solution integrates speech transcription, RAG, LLM, and text-to-speech components, with Cerebras powering the inference for both customer support and real-time compliance evaluation agents.
  47. Alex - iOS Development with AI and Cerebras InferenceThis post details the integration of Cerebras Inference into the Alex iOS development toolkit. This integration led to a 15x latency improvement for the 'Code Apply' feature, reducing execution time from fifteen seconds to less than a second by enabling precise application of AI-generated snippets. Additionally, Cerebras Inference was used to summarize web search results, reducing context size by 90%, which improved speed and reduced costs for developers accessing up-to-date coding insights.
  48. Compressing KV cache memory by half with sparse attentionThis post introduces techniques to compress KV cache memory by half in LLMs using sparse attention. It details the application of synthetic long instruction data generation for fine-tuning, selective dense layers based on sensitivity analysis, specialized sliding-window attention with global sink tokens, and auxiliary memory tokens. The authors release two models (Llama-3-CBHybridL-8B and Llama-3-CBHybridM-8B) and the synthetic dataset used for fine-tuning. The contribution focuses on the methods for selecting sensitive layers for dense attention and the use of memory tokens to enhance long-context performance.
  49. Cerebras Launches World Fastest DeepSeek R1 Llama-70B Inference - CerebrasThis post announces the deployment of DeepSeek R1 Llama-70B on Cerebras Inference, achieving over 1,500 tokens/s, which is 57x faster than GPU solutions. It highlights the reduction of reasoning prompt time from minutes to seconds (e.g., 22 seconds on o1-mini to 1.5 seconds on Cerebras for a coding prompt). The post emphasizes the practical benefits of this speedup for coding, research, and agent applications, and reiterates Cerebras's commitment to data privacy and security with US-based infrastructure and zero data retention.
  50. Extending LLM context with 99% less training tokens - CerebrasThis post presents a new recipe for extending LLM context length with significantly reduced training tokens (~99% reduction compared to full context training). It details the use of Retrieval Augmented Fine-Tuning (RAFT) with distractor passages, syntactic question generation, and translation for data augmentation. It also incorporates train-time token position ID shifting (simplified CREAM) and RoPE position extrapolation with an adjusted base frequency. The training pipeline consists of CPT@32K, SFT@8K, and SFT@32K stages, with specific data mixes and hyper-parameter settings provided for each.
  51. Dolphin - Efficient Data Curation at Cognitive Computations with Cerebras - CerebrasThis post details how Cerebras inference was used to accelerate data curation for training the Dolphin LLM models. It highlights a 90% reduction in processing time and a 10x increase in tokens per second (from 100 to 1,000), enabling the processing of millions of samples in two days instead of a month. The system's ability to handle concurrent requests improved from 20 to 200 requests per minute. The post emphasizes the rapid, nuanced classification of complex conversation samples within milliseconds to filter out unwanted content, contributing to improved AI model quality and faster iteration cycles.
  52. Visualizing Knowledge at Lightning Speed with Cerebras at Tako - CerebrasThis post details how Tako integrated Cerebras's inference API to run the Llama 3.3-70B model, achieving a 60% reduction in median and P95 search latency. This performance improvement enabled Tako to generate knowledge cards at scale with large datasets in near real-time, unlocking new features like Related Search and improving the core search product experience.
  53. Cerebras brings instant inference to Mistral Le Chat - CerebrasCerebras Inference is now powering Mistral's Le Chat platform, specifically its new Flash Answers feature. This integration achieves over 1,100 token/s for the Mistral Large 2 model, a 10x improvement over popular models. The performance is attributed to the Wafer Scale Engine 3's SRAM-based inference architecture and speculative decoding techniques developed in collaboration with Mistral researchers. The initial release focuses on serving text-based queries.
  54. Building Real Time Digital Twin with Cerebras at Tavus - CerebrasTavus integrated Cerebras's inference engine into their Conversational Video Interface (CVI) pipeline to reduce LLM latency. This resulted in a 66% reduction in Time to First Token (TTFT) from 300ms to 100ms and a 300% increase in Token Output Speed (TPS) from 1/30ms to 1/10ms for the Llama 3.1-8B model. These optimizations led to an overall LLM latency reduction of 100-1000%, enhancing the realism and responsiveness of their digital twin video experiences.
  55. CePO: Empowering Llama with Reasoning using Test-Time Compute - CerebrasIntroduces CePO (Cerebras Planning and Optimization), a framework that enhances Llama models with reasoning capabilities via test-time computation. CePO enables Llama 3.3-70B to surpass Llama-3.1 405B in accuracy on coding, math, and reasoning benchmarks, achieving ~100 tokens/second on Cerebras hardware. The framework's pipeline includes step-by-step planning, multiple response generation, inconsistency analysis, and a Best-of-N selection with confidence scoring. CePO is compared against CoT with Reflection, Self-Consistency, and Best-of-N prompting strategies, demonstrating consistent improvement across various tasks.
  56. Memo-ry: Simplifying Daily Tasks for People with Memory Loss - CerebrasThe Memo-ry application demonstrates the practical application of Cerebras's fast inference technology for real-time task extraction from speech. It highlights a reduction in latency from ~300ms to under 50ms, a 85% efficiency improvement, which is critical for maintaining conversational flow and usability for individuals with cognitive difficulties. The architecture integrates frontend, speech recognition (Web Speech API), Cerebras inference for task extraction, and dynamic UI updates.
  57. AIBI: Revolutionizing Interviews with AI (AI interviews aren't really a thing yet) - CerebrasThis post details the implementation of AIBI, an AI Bot Interviewer, which utilizes Cerebras inference to achieve significant latency reductions (72-75%) for Llama70B models. This improvement enables real-time technical question generation, audio interview conduction, and candidate evaluation, drastically enhancing the candidate experience compared to previous GPT-4o implementations. The post breaks down the architecture into an HR Dashboard Platform and an AI Interviewer Bot, detailing latency improvements at each stage, including job application form generation, preliminary question generation, and the interview process itself (question asking, candidate response transcription, and answer generation).
  58. CerebrasIntroduces LongCePO (Long-Context Cerebras Planning and Optimization), an extension of the CePO framework. LongCePO enables LLMs to effectively manage and process information beyond their inherent context windows by employing strategic planning and iterative steps to access and integrate relevant information from extended data sources. This allows for "infinite context" at inference time, achieving frontier-level performance on long-context tasks (e.g., LongBench v2) with limited model context windows (~8K tokens), and outperforming models like Mistral-Large-Instruct-2411 and o1-mini-2024-09-12. The framework will be open-sourced.
  59. Llama3.1 Model Quality Evaluation: Cerebras, Groq, SambaNova, Together, and Fireworks - CerebrasThis post benchmarks the Llama3.1-8B and Llama3.1-70B Instruct models on Cerebras inference hardware against competitors (Groq, SambaNova, Together, Fireworks). It details rigorous evaluations across general reasoning (MMLU, MATH, GPQA, DROP, MGSM), coding (HumanEval, MBPP, HumanEval+, MBPP+), and multi-turn conversations (MT-Bench). Cerebras' Llama3.1-70B model is shown to outperform competitors on 9 out of 10 benchmarks, demonstrating leading accuracy and efficiency. The post emphasizes the importance of comprehensive, apples-to-apples benchmarking to ensure model quality, especially when techniques like quantization are used.
  60. Building an AI-Powered Search Assistant for Zoom Team Chat - CerebrasThis post details the integration of Cerebras inference with the Zoom Developer Platform and Exa's neural search to build an AI-powered search assistant for Zoom Team Chat. The bot intelligently decides when to search for current information using Cerebras, processes search results, and generates contextual responses with minimal latency. It also maintains conversation history for better context. The integration leverages Cerebras for fast inference and response generation, Exa for neural search, and the Zoom Developer Platform for bot interaction. The post also outlines the authentication flow for the Zoom Chatbot using OAuth 2.0 Client Credentials grant type.
  61. Chatting Your Way Through 4500 NeurIPS papers with Cerebras - CerebrasThe post details the development of a web application for NeurIPS 2024 papers that uses Cerebras Inference and PostgreSQL's full-text search for RAG. It highlights the challenge of RAG latency bottlenecked by inference speed and describes the process of indexing PDFs into a vector database using LlamaIndex, text extraction, chunking, metadata enrichment, and embedding creation with BAAI's bge-large-en-v1.5 model. The use of Supabase for vector storage and pgvector for similarity search is explained. A key contribution is the reported 17x speed-up in response time by switching from GPU-based providers to Cerebras for LLM inference in the RAG pipeline, reducing average answer time from 8.5 seconds to under 2 seconds.
  62. Introducing Cerebras Inference: AI at Instant Speed - CerebrasThis post introduces Cerebras Inference, highlighting its ability to deliver 1,800 tokens/sec for Llama3.1 8B and 450 tokens/sec for Llama3.1 70B, which is claimed to be 20x faster than NVIDIA GPU-based hyperscale clouds and 2.4x faster than Groq for Llama3.1-8B. The core technical innovation is the Cerebras Wafer Scale Engine (WSE-3) which integrates 44GB of SRAM on-chip, providing 21 petabytes/s of aggregate memory bandwidth. This eliminates the need to move model parameters from external memory for each token generation, a key bottleneck in GPU inference. The post explains that for models exceeding single-wafer capacity, they are split at layer boundaries and mapped to multiple CS-3 systems. It also emphasizes the use of native 16-bit model weights for accuracy, contrasting with 8-bit quantization used by some competitors. The Cerebras Inference API is available via chat and API access, built on the OpenAI Chat Completions format, with initial free token offerings and competitive pricing for scaled deployments. The post also touches on how fast inference enables new techniques like scaffolding for more thoughtful AI responses.
  63. Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s - CerebrasThis post details a 3x performance improvement for Cerebras Inference, achieving 2,100 tokens/sec for Llama3.1-70B. This was achieved through re-writing and optimizing critical kernels (MatMul, reduce/broadcast, element-wise ops, activations), streamlining wafer IO to run asynchronously from compute, and implementing speculative decoding. The post also highlights the performance advantages in terms of output speed per user, time to first token, and total response time compared to GPU-based solutions and hyperscale clouds.
  64. Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference - CerebrasAchieved 969 output tokens/s for Llama 3.1 405B, a 12x improvement over best GPU results. Demonstrated 240ms time-to-first-token latency. Supported 128K context length with high performance. Showcased 16-bit weights for full model accuracy. Achieved 539 tokens/s with a 100,000 token prompt, outperforming competitors significantly. Detailed pricing for Q1 2025 availability.
  65. BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model - CerebrasIntroduces BTLM-3B-8K, a 3 billion parameter language model that achieves 7 billion parameter performance, optimized for long sequence length inference (8K+ tokens). It demonstrates the ability to run on devices with as little as 3GB of memory when quantized to 4-bit. The model was trained on the SlimPajama dataset and utilizes ALiBi position embeddings with variable sequence length training for improved extrapolation. The training process incorporated maximal update parameterization, improved hyperparameter tuning, an updated model architecture, extensive data cleaning and deduplication, and variable sequence length training. The post highlights the training on the CG-1 AI supercomputer, emphasizing the Cerebras weight streaming architecture's scalability and simplicity, and notes zero hardware failures during training.
  66. Variable Sequence Length Training for Long-Context Large Language Models - CerebrasIntroduces Variable Sequence Length (VSL) training for LLMs. VSL stages training into short (e.g., 2k) and long (e.g., 8k) sequence phases. This method reduces FLOPs by up to 29% compared to training with long sequences throughout, without requiring architectural changes or hyperparameter tuning. It demonstrates comparable model performance and generalizes across different position embeddings (NoPE, learned, RoPE, ALiBi). The Cerebras Software Platform (CSoft) simplifies VSL implementation by modifying data subset paths.
  67. Context is Everything: Why Maximum Sequence Length Matters - CerebrasThis post explains the quadratic complexity of attention layers in transformer models, where memory and compute grow by a factor of 4 for every 2x increase in sequence length. It contrasts this with previous attempts to reduce complexity (sparsification, hierarchy, approximation) which introduce presumptions about data patterns. The post then details how Cerebras' Wafer-Scale Engine (WSE), with its 850,000 PEs, ultra-low latency fabric, and weight streaming feature, enables processing of sequences up to 50,000 tokens with dense attention, a 20x increase over modern GPUs. This is achieved by keeping activations on the wafer and loading weights layer by layer, freeing up memory and leveraging on-chip network bandwidth. The post also presents results showing improved next token prediction accuracy with longer sequences and highlights potential applications in document summarization, drug discovery, and genomics.
  68. Innovations in Virtual Drug Screening - CerebrasThis post details how Cerebras CS-2 systems accelerate virtual drug screening by over 50x, reducing screening time from 183 days to 3.5 days. It explains that traditional GPUs are inefficient for the small, fully connected models used in drug screening due to low utilization and memory bottlenecks. The Cerebras WSE architecture, with its 850,000 AI-optimized cores, high-speed memory, and low-latency fabric, allows for efficient mapping and distribution of computations for multiple copies of these small models, achieving massive inference acceleration and high throughput.
  69. An AI Chip With Unprecedented Performance To Do the Unimaginable - CerebrasThis post introduces the Wafer Scale Engine 2 (WSE-2), the largest AI accelerator chip with 850,000 cores on a single wafer, providing 220Pb/s of aggregate fabric bandwidth and 40GB of on-wafer memory. It highlights the system's ability to harvest sparsity in computations, improving power efficiency and performance. The uniform fabric across the wafer simplifies programming and optimizes time-to-solution by allowing seamless mapping of problems without regard to physical placement. The architecture enables high utilization and sustained throughput for large models by overlapping compute and communication effectively.
  70. Building Real-Time AI-Powered Contact Centers with Cerebras at OpenCall - CerebrasThis post details how OpenCall integrated Cerebras's LLM inference API to significantly reduce latency in their AI-powered contact center. By breaking down complex tasks into atomic operations and leveraging Cerebras for fast and reliable inference, OpenCall achieved a 90% reduction in overall latency, enabling real-time processing of complex workflows and leading to a substantial increase in customer satisfaction scores.
  71. Guii: An Interactive Coding Companion for Creative Frontend Development - CerebrasIntroduces Guii, an interactive coding companion for frontend development that integrates visual design and code modification. It leverages Cerebras inference for near-instantaneous feedback (2150 tokens/sec for Llama3.1-70B, a 16x speedup over previous providers), enabling developers to select visual elements on a webpage and prompt for real-time code changes. The system analyzes codebase structure and integrates with build tools like Vite for instant previews. It also discusses potential future extensions to art creation and game development.
  72. Simulating Human Behavior with Cerebras - CerebrasThe post details the application of Cerebras hardware for simulating human behavior at scale using a multi-LLM framework called LlamaSim. It highlights the achievement of over 2000 tokens/sec for Llama 3.1 8B on Cerebras, significantly outperforming other providers for tasks like generating synthetic identities and backstories. It also mentions the use of Cerebras integration with Instructor for structured output generation and the challenges of managing conversation context with limited context windows.
  73. The Cerebras AI Model Studio brings Wafer-Scale Cluster Acceleration to the Cloud - CerebrasThis post introduces the Cerebras AI Model Studio, a cloud service powered by the Cerebras Wafer-Scale Cluster, designed to make training large language models cheaper, faster, and easier. It addresses the high cost, time, and complexity of training LLMs by providing accessible cloud clusters of Cerebras CS-2 systems. The service offers model-based pricing for GPT-style models, predictable costs, and pre-configured Python scripts for training various GPT and T5 models. It also emphasizes security through partnership with Cirrascale and offers add-ons for increased resources, cluster time, and inference support.
  74. Cerebras Announces Fine-Tuning on the Cerebras AI Model Studio - CerebrasThis post announces the launch of fine-tuning capabilities on the Cerebras AI Model Studio, enabling users to fine-tune models like GPT-J, GPT-NeoX, and CodeGen. It highlights competitive pricing ($0.00096 per 1000 tokens for GPT-J 6B) and performance improvements (8x faster than traditional cloud for GPT-NeoX 20B with 10B tokens). The post contrasts fine-tuning with pre-training and discusses challenges in fine-tuning large models at scale with traditional systems, emphasizing Cerebras's advantages in speed, cost, and ease of use through its Wafer-Scale Cluster. It also assures users of data security and ownership of trained weights.
  75. Cerebras and Qualcomm Unleash ~10X Inference Performance Boost with Hardware-Aware LLM Training - CerebrasThis post details hardware-aware LLM training and fine-tuning techniques to optimize inference performance on Qualcomm Cloud AI 100 accelerators. It introduces Micro-scaling (Mx) formats for model parameter compression, speculative sampling using a draft LLM model, and unstructured sparsity for model compression. These techniques, when combined, aim to deliver up to 10x acceleration. The post also highlights the collaboration between Cerebras and Qualcomm in enabling these advanced techniques from training to inference.
  76. ReadAgent: Bringing Gist Memory to AI - CerebrasThis post details the implementation of Google DeepMind's ReadAgent workflow using Cerebras's Inference SDK. It breaks down the ReadAgent process into three steps: Episode Pagination, Memory Gisting, and Interactive Lookup. The post emphasizes the critical role of low-latency LLM inference for the efficiency of ReadAgent's iterative API calls and discusses how faster inference enables advanced methods like chain-of-thought reasoning. The Cerebras implementation showcases the integration of agentic workflows with fast inference capabilities.
  77. Introducing DocChat: GPT-4 Level Conversational QA Trained In a Few Hours - CerebrasCerebras DocChat models (Llama3-DocChat and Dragon-DocChat) are released for document-based conversational QA. Llama3-DocChat was trained in hours on a single Cerebras System, and Dragon-DocChat was fine-tuned in minutes. The training process incorporated synthetic data generation to handle unanswerable questions, Chain of Thought reasoning for arithmetic tasks, and instruction tuning for entity extraction. The models, training recipes, and datasets are open-sourced.
  78. UndatedAccelerating GPT-5.6 Sol Ultrafast with OpenAIThis post introduces Ultrafast Mode, a new service tier for OpenAI API powered by Cerebras, which accelerates GPT-5.6 Sol to deliver up to 750 output tokens per second. It highlights performance improvements over existing models on benchmarks like Humanity's Last Exam and GDP-Val, demonstrating a 7x speedup on HLE and a 5.6x end-to-end speedup on GDP-Val with no quality degradation. The technical innovation enabling this speed is Cerebras's Wafer-Scale Engine architecture, which packs 44GB of SRAM on-chip to keep model weights local and minimize data movement, contrasting with GPU-based approaches bottlenecked by off-chip storage transfers.