Kimi 发布 WorldVQA 基准，3500 组图文题检验多模态模型的视觉事实可靠性
WorldVQA
- 1Kimi 团队发布 WorldVQA 基准，用于测量多模态大模型对所见实体的事实性识别是否可靠。
- 2数据集含 3500 组经多阶段人工核验的图像问答对，覆盖 9 类知识并区分常见与长尾分布。
- 3官方实验称前沿模型在长尾视觉知识上准确率常低于 50%，且各模型普遍呈过度自信倾向。
- 4Kimi-K2.5 以 ECE 37.9% 和 Slope 0.550 在两项校准指标上表现最好，数据与脚本已开源。
这个基准补上了模型认错视觉实体却依旧自信的度量缺口，官方数据显示前沿模型的诚实度距理想值还很远。
- 影响：模型在长尾视觉知识上准确率常低于 50%，直接用于识别类业务存在出错风险；模型普遍过度自信，置信度分数不能当作可靠的质量过滤手段；数据集与评测脚本开源，企业自测的门槛在降低。
- 参考：可考虑用开源数据集抽测与自家业务相关的类别，记录候选模型的错误率和置信度偏差；值得先盘点现有视觉流程里哪些环节依赖模型自知出错的能力。
- 依据：全部结果来自 Kimi 官方博客的自测口径，暂无第三方独立验证；Kimi-K2.5 的 ECE 37.9% 与 Slope 0.550 为官方实验数字。
Introducing WorldVQA
A benchmark for evaluating atomic visual world knowledge in Multimodal LLMs.
Authors Kimi Team
Overview
We are releasing WorldVQA, a new benchmark designed to measure the factual correctness of Multimodal Large Language Models (MLLMs). While recent models have demonstrated impressive capabilities in visual reasoning and description, measuring their reliability regarding visual world knowledge remains a challenge.
WorldVQA focuses on a critical question: Does the model actually recognize the specific entity it sees, or is it merely hallucinating based on visual patterns?
Our results show that WorldVQA creates a significant challenge for frontier models. Even state-of-the-art models struggle to achieve high accuracy on long-tail visual knowledge, often falling below 50% accuracy. This benchmark aims to drive progress toward more factually reliable and knowledgeable multimodal AI.
The Dataset
The dataset consists of 3,500 high-quality image-question pairs. The distribution aims to test a model's encyclopedic breadth across the world. The dataset distinguishes itself through three core design principles:
- Factuality & Unambiguity: Every question has a single, verifiable ground-truth answer. We exclude subjective questions or ambiguous visual scenarios.
- Rich Taxonomy: The dataset spans 9 categories to ensure broad coverage of world knowledge.
- Head vs. Tail Distribution: We explicitly separate data into Head (common knowledge) and Tail (rare/long-tail knowledge). This allows us to measure how model performance degrades as knowledge becomes more obscure.
Note on Quality: To ensure the benchmark is a reliable gold standard, all images and question-answer pairs underwent rigorous multi-stage human verification to filter out noise and ambiguity.
Distribution of Tasks per Category
| Statistics | Number |
|---|---|
| Data | |
| Total | 3500 |
| Chinese (CN) | 1260 (36%) |
| English (EN) | 2240 (64%) |
| Category Categories | |
| Total categories | 9 |
| Nature & Environment (Nature) | 9.31% |
| Locations & Architecture (Geography) | 14.63% |
| Culture, Arts & Crafts (Culture) | 14.46% |
| Objects & Products (Objects) | 12.49% |
| Vehicles, Craft & Transportation (Transportation) | 8.74% |
| Entertainment, Media & Gaming (Entertainment) | 14.60% |
| Brands, Logos & Graphic Design (Brands) | 7.43% |
| Sports, Gear & Venues (Sports) | 4.06% |
| Notable People & Public Figures (People) | 14.29% |
| Difficulty | |
| Easy | 31.16% |
| Medium | 40.77% |
| Hard | 28.07% |
Using WorldVQA to compare models
Measuring Calibration: Confidence vs. Accuracy
In our experiments comparing model confidence with actual accuracy, we utilized two key metrics to measure the alignment between a model's subjective belief and its objective performance:
- ECE (Expected Calibration Error): Measures the average gap between the model's subjective confidence and its objective accuracy. The ideal value is 0.
- Slope (Weighted Average Slope): Measures the correlation and sensitivity between the model's accuracy and its own confidence. The ideal value is 1.0.
Calibration and Confidence Distribution Analysis. Left: Reliability diagrams plotting Actual Accuracy against Stated Confidence. To ensure statistical significance, only bins containing more than 20 samples are visualized. The size of each data point is proportional to the number of samples in that bin. The black dashed diagonal (y=x) represents perfect calibration, while colored dashed lines indicate the weighted average slope for each model. Right: The distribution of stated confidence scores across the full dataset (without sample thresholding). The plots reveal a severe overconfidence trend, with most models concentrating their predictions in the 90-100% confidence range.
Our experiments reveal that all evaluated models are currently far from the ideal state, exhibiting a universal tendency toward overconfidence.
While Kimi-K2.5 achieves best performance on both metrics—recording an ECE of 37.9% and a Slope of 0.550—there remains a significant gap to bridge in the pursuit of "honesty" and "alignment." Enhancing the self-awareness boundaries of multimodal models represents a critical direction for future exploration.
Conclusion
WorldVQA is a simple but challenging benchmark for evaluating the atomic visual knowledge of frontier models. Improving performance on WorldVQA is a necessary step for the next generation of AI agents. We are open-sourcing the WorldVQA dataset and evaluation scripts to help the community address the visual knowledge gap.
- Read the Paper: https://arxiv.org/abs/2602.02537
- View the Code: https://github.com/MoonshotAI/WorldVQA
- Download the Data: https://huggingface.co/datasets/moonshotai/WorldVQA
来源：Kimi：研究博客 · kimi.com