Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta

This talk details a system for analyzing short-form video at scale, addressing two core problems: modality misalignment within a single video (e.g., a video changing topic abruptly) and the attribution of unoriginal content. The solution employs a multi-agent architecture: a 'perceiver' agent splits videos based on temporal change, emitting clip-level embeddings, tags, and OCR; a 'reviewer' agent analyzes this data along with live comments and sentiment; and a 'retriever' agent indexes topics, embeddings, and entities into inverted, vector, and graph stores for similarity searches. Specialized VLMs are pretrained on in-house user-generated content and instruction-tuned to a JSON schema, with DPO loops for refinement. Evaluation includes per-node latency, reasoning budgets, and token cost. Optimizations like frame compression, verdict caching, and metadata pruning reduce pipeline load.