From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam

This post details the training of a 3 billion parameter state-space model (SSM) for vision-language tasks, specifically for Indic document intelligence. Key contributions include: 1) A contrarian architectural choice of SSM over transformers to handle long visual sequences efficiently. 2) A novel training curriculum that starts with 13 trillion text tokens across multiple languages before introducing visual data, enabling a strong language prior. 3) A 'block-level OCR wrapped in a layout harness and a reading order harness' approach, diverging from monolithic page-level VLMs. 4) The use of verifiable reward reinforcement learning, with rewards based on unit tests for character error rate, table structure, and grammar, making RL scalable for OCR. 5) The development of a data engine for low-resource languages and evals built for practical usefulness. The resulting model achieves SOTA performance and is in production digitizing millions of pages.