How we built a custom vision LLM to improve document processing at Grab

Developed a custom lightweight Vision LLM (~1B parameters) from scratch by combining the vision encoder from Qwen2-VL 2B and the language decoder from Qwen2.5 0.5B, connected by an adjusted projector layer. Implemented a four-stage training process: projector alignment, vision tower enhancement on public multimodal datasets, language-specific visual training with synthetic OCR data, and task-centric fine-tuning. Achieved performance comparable to the larger 2B model with significantly reduced latency, outperforming traditional OCR and external APIs.