TechPrufer
Get started
AI Engineer
AI Engineer logoAI Engineer · Feature trail

Agentic Document Intelligence and OCR Correction

5 videos · Sep 2026

What this trail covers

This post contributes to the field of agentic document intelligence by detailing how to build robust AI systems that can process and reason over complex, non-standard document formats like PDFs. It highlights the challenges of traditional OCR and RAG when dealing with visual encoding of meaning, and introduces agentic approaches for improved accuracy and reasoning. The work emphasizes the importance of specialized data processing, agent harnesses for multi-step problem-solving, and comprehensive evaluation strategies. The Large Taste Model at Spotify, an LLM-native system, demonstrates generative personalization across a vast catalog of music and podcasts, moving from curated playlists to steerable and editable user experiences. The training methodology, NEO, involves domain grounding with a frozen backbone and multitask tuning, which significantly improves performance on various Spotify tasks, including cold-starting audiobook recommendations. Furthermore, grounded LLM judges, informed by user profiles and behavior, show a substantial increase in agreement with human judgments on ambiguous queries, enhancing evaluation reliability.

All AI Engineer trails
  1. Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, SpotifyThis talk details Spotify's development of the Large Taste Model, an LLM-native system for generative personalization. It introduces the NEO training recipe, which involves adding semantic IDs to an open-weight LLM (like Qwen) and grounding these new tokens while keeping the backbone frozen. This approach preserves the LLM's general language abilities, unlike continued pre-training. The model is then instruction-tuned across various Spotify tasks, leading to improvements like cold-starting audiobook recommendations. The talk also discusses decoding strategies and how grounding LLM judges with user profiles and behavior increased their agreement with humans by 91% on ambiguous queries, improving evaluation set scaling.
  2. One Operator, Many Drones: Inside Skydio's Autonomy Stack — Suchet Bargoti, SkydioThis talk details Skydio's autonomy stack for managing a fleet of drones, emphasizing the split intelligence between edge and cloud, the use of maps as world models updated by the fleet, object tracking through occlusion, and the application of VLM agents with tool calls for tasks like "find and follow." It also discusses the limitations of end-to-end learning for physical systems requiring high reliability, contrasting it with agentic approaches for specific tasks. The core contribution is the engineering of a distributed, intelligent drone fleet system that leverages AI for complex operational tasks.
  3. From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, ReductoThis talk introduces agentic OCR that performs token-level corrections (e.g., changing an 'O' to a '0') rather than full page rewrites, significantly improving accuracy and efficiency. It also details how to handle tables by converting simple ones to markdown and complex ones to HTML, and uses natural language rendering for retrieval to overcome embedding model limitations. The post discusses classification and splitting for orchestrating large documents and the use of agent harnesses with tools like code interpreters and self-checks to solve complex problems (e.g., converting line charts to data tables) that frontier models struggle with, addressing precision and recall trade-offs. It stresses the importance of evaluations at every stage and envisions agents interacting with file systems and CLIs.
  4. Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, MetaThis talk details a system for analyzing short-form video at scale, addressing two core problems: modality misalignment within a single video (e.g., a video changing topic abruptly) and the attribution of unoriginal content. The solution employs a multi-agent architecture: a 'perceiver' agent splits videos based on temporal change, emitting clip-level embeddings, tags, and OCR; a 'reviewer' agent analyzes this data along with live comments and sentiment; and a 'retriever' agent indexes topics, embeddings, and entities into inverted, vector, and graph stores for similarity searches. Specialized VLMs are pretrained on in-house user-generated content and instruction-tuned to a JSON schema, with DPO loops for refinement. Evaluation includes per-node latency, reasoning budgets, and token cost. Optimizations like frame compression, verdict caching, and metadata pruning reduce pipeline load.
  5. Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging FaceThis talk advocates for using coding agents with pre-trained, Apache 2.0 licensed models instead of directly deploying large Vision Language Models (VLMs) for real-time computer vision tasks. It introduces a toolkit that leverages a coding agent to act as a 'clueless computer vision engineer.' This agent uses a large VLM for initial labeling (vibe training) on unlabeled datasets, then employs two smaller VLM judges to refine bounding boxes based on minimum agreement, and finally trains a more efficient RF-DETR model. The process is demonstrated to be cost-effective (a few dollars) and achieves strong results on road sign detection and document parsing, even identifying signatures missed by the initial labeling VLM. Key findings highlight judge imbalance, the need for human approval of judge prompts, and agent blunders like flipping traffic signs. The toolkit also includes smaller models for segmentation, pose, depth, and OCR, with future plans for image-guided detection.