The Problem: While Multimodal Large Language Models (MLLMs) are getting better at processing video, the industry lacks a comprehensive way to test their true capabilities. Existing benchmarks only evaluate pieces of the puzzle. If an AI summarizes a video or answers a question about it, it’s hard to know if it actually understands the context or if it's just making an educated guess. Without a rigorous, systematic test, developers can't diagnose whether an AI actually knows
where and
when something happened across complex mixtures of sights, sounds, and text.
The Breakthrough: The researchers introduce
OmniCompass, a holistic benchmark designed to stress-test AI on "omni-modal" video understanding. It challenges models to answer three exact questions about a video:
When did a specific target appear (temporal grounding)?
Where is it on the screen (spatial grounding)? And
What happened (question answering)? The test covers diverse video lengths and splits into two tracks: one for Vision-Language (visual evidence) and one for Audio-Visual (joint sight and sound evidence). To ensure true understanding, it even tests models twice on the exact same target - once using just text, and once using text paired with an image.
Key Findings: After testing 50 different AI models, the researchers discovered that open-source models still heavily lag behind paid, proprietary models - especially in audio-visual integration. Crucially, even the most advanced leading models struggle to precisely pinpoint the exact time and location of an object. Furthermore, models are brittle: their predictions often become inconsistent depending on whether the target was specified by text alone or with an image.
Why This Matters: You can't improve what you can't measure. For builders and executives, OmniCompass provides the definitive ruler needed to expose current blind spots in video AI. Pinpoint accuracy in video is notoriously hard, and this benchmark proves that even the best models aren't quite ready for mission-critical tasks that require exact timing or spatial awareness.
Business Impact: This paper provides a vital roadmap for companies building the next generation of video AI products. Whether you are developing smart security systems, automated sports highlight generators, media compliance tools, or advanced video-search platforms, understanding these exact model limitations helps prevent costly deployment errors. OmniCompass gives startups and enterprises a clear standard to validate that their video agents actually understand the
what, where, and when of real-world media.
Generated by Gemini