Ulysses Sequence Parallelism: Training with Million-Token Contexts
Jul 26, 2025
Summary
This post introduces Ulysses Sequence Parallelism, a novel technique for training large language models with million-token contexts. It details how sequence parallelism is applied to manage memory and computation, enabling the training of models like Qwen/Qwen3-4B with significantly extended context windows. The implementation focuses on efficient communication strategies and memory management to overcome the challenges associated with processing extremely long sequences.
More from the Long Context Window Models trail
- GLM-5.2: Built for Long-Horizon Tasks
- Introducing Mellum2: A 12B Mixture-of-Experts Model by JetBrains
- DeepSeek-V4: a million-token context that agents can actually use
- Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality
- Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents