Kimi 团队开源 PerceptionBench 基准，16 个前沿多模态模型均未达 60% 准确率
PerceptionBench
- 1Kimi 团队发布 PerceptionBench，从 42 个基准的模型失败案例归纳出 10 项原子视觉感知能力并开源
- 2基准含 3,000 道已验证题目，覆盖 10 类感知能力，作答只靠看图，不需推理或外部知识
- 316 个前沿多模态模型无一准确率达到 60%，感知相关幻觉是平均表现最弱的能力
- 4题目 60% 归因自模型真实失败的拆解、40% 为新作，内部已验证题库超 17,000 道
多模态模型的视觉感知仍是普遍短板，总分相近的模型实际看图能力可能差距很大，综合跑分不足以支撑选型判断。
- 影响：涉及 OCR、图像审核等强感知场景的错误率可能高于综合跑分给出的预期；幻觉是平均最弱项，高风险环节需保留人工复核；模型选型宜按感知能力维度做拆分评测。
- 参考：可考虑用开源基准对候选多模态模型做一轮能力拆分测试，对照自身业务的错误类型清单；值得把幻觉与细粒度识别的单项成绩纳入试点准入条件。
- 依据：结论来自 Kimi 团队对 16 个前沿模型的自建基准测试，属发布方官方口径，暂无第三方独立复测。
Introducing PerceptionBench
Evaluating Atomic Visual Perception in Multimodal Large Language Models
Authors Kimi Team
Overview
We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities—discovered from how today's models fail, not defined in advance. From frontier-model failures across 42 benchmarks, we derive 10 atomic perceptual capabilities and construct 3,000 verified questions, each isolating a single capability and answerable by looking, with no reasoning or external knowledge required.
Across sixteen frontier MLLMs, no model reaches 60% accuracy, and perception-related hallucination is the weakest capability on average. Models with nearly identical overall scores can diverge sharply in what they actually perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently.
The Dataset
Guided by the induced taxonomy, we select the most informative failures from the source benchmarks, decompose them into atomic sub-questions, and author additional questions on supplemented images. The retained and constructed samples together form an in-house pool of 17,000+ verified questions. The released benchmark subsamples 3,000 verified questions from the constructed portion—60% decomposed from attributed model failures and 40% newly authored—with category-level balancing and difficulty stratification to isolate atomic perceptual capabilities from confounding factors. The released benchmark distinguishes itself through three core design principles:
- Failure-Driven Taxonomy: Every category is discovered from real model failures, attributed to the earliest erroneous step across 42 existing benchmarks.
- Ten Atomic Perceptual Categories: Visual Relation, Counting, Attribute, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
- Perception, Not Reasoning or Knowledge: Samples are curated, decomposed, and difficulty-balanced so that difficulty stems from perception rather than reasoning or external knowledge.
Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views.
Distribution of Tasks per Category
| Statistics | Number |
|---|---|
| Data | |
| Total | 3,000 |
| Atomic perceptual categories | 10 |
| Task Categories | |
| Depth 3D Perception Error | 330 (11.00%) |
| Visual Counting Error | 330 (11.00%) |
| Fine-Grained Recognition Error | 290 (9.67%) |
| Visual Relation Error | 330 (11.00%) |
| Visual Attribute Error | 330 (11.00%) |
| Visual Localization Error | 330 (11.00%) |
| Visual Comparison Error | 279 (9.30%) |
| Context Integration Error | 255 (8.50%) |
| Hallucination | 271 (9.03%) |
| OCR Error | 255 (8.50%) |
Using PerceptionBench to Compare Models
Conclusion
PerceptionBench is a simple but challenging benchmark for evaluating the atomic visual perception of frontier models. It provides a capability-level standard for measuring and diagnosing the visual perception boundaries of multimodal models. We are open-sourcing the PerceptionBench dataset and evaluation code to help the community address the visual perception gap.
- Download the Data: https://github.com/MoonshotAI/PerceptionBench
来源：Kimi：研究博客 · kimi.com