TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

Looped Language Models

3 posts · Dec 2025 – Apr 2026

What this trail covers

Together AI continues to explore and enable advanced model architectures like Parcae, which introduce stable looped language models. This approach allows for achieving higher quality with fewer parameters by increasing recurrence rather than solely scaling data, opening efficient frontiers for training memory-constrained on-device models. The research focuses on stabilizing training for these looped architectures, establishing new scaling laws, and demonstrating their effectiveness in matching or exceeding the performance of larger fixed-depth models. Together AI is releasing training code and models for Parcae to accelerate research in parameter efficiency.

All Together AI trails
  1. Parcae: Doing more with fewer parameters using stable looped modelsThis post introduces Parcae, a stable architecture for looped language models that achieves the quality of a Transformer twice its size with predictable training. It addresses instability issues in prior looped models by analyzing their linear dynamical system properties and constraining the spectral radius of the recurrent block's update matrix. Parcae demonstrates improved validation perplexity and downstream task performance compared to parameter- and data-matched Transformers and prior looped models (RDMs). The post also establishes the first scaling laws for looping, showing that compute-optimal training requires increasing looping and data in tandem, and that looping creates a stricter Pareto Frontier for validation loss. Training code and models for Parcae are released.
  2. CoderForge-Preview: SOTA open dataset for training efficient coding agentsThis post introduces CoderForge-Preview, the largest open dataset of coding agent trajectories (258k test-verified trajectories). It details the data generation process using Qwen3-Coder-480B, including task sources (R2E-Gym, SWE-Smith, SWE-Rebench), setup with OpenHands scaffold, and filtering for test-verified trajectories. The post compares CoderForge-Preview to other datasets, highlighting its scale and long-context coverage. It also analyzes trajectory success rates, data generation costs, trajectory characteristics (length, steps), and license filtering. Furthermore, it describes the training setup for fine-tuning Qwen3-32B using sequence parallelism (Ulysses), multi-packing, and a token-level loss formulation, achieving 59.4% pass@1 on SWE-Bench Verified.
  3. Together AI and Meta partner to bring PyTorch Reinforcement Learning to the AI Native CloudThis post announces a partnership between Together AI and Meta's PyTorch team to build an open-source reinforcement learning framework for training agentic AI systems. It highlights the growing importance of RL for modern agentic models that need to operate over longer time horizons and interact with dynamic environments. The integration will bring sophisticated, high-performance RL capabilities to the Together AI platform, complementing existing services and providing a complete solution for building, training, and deploying agentic systems.