Artificial Analysis 新版指数扩大开源与闭源模型差距

New Artificial Analysis index widens the gap between open and closed models

文章要点

1Artificial Analysis 发布第 5 版 Intelligence Index，开源模型与榜首的分差从 3 分扩大到 12 分。
2Kimi K3.1 Pro 在新版指数中从第 4 名跌至第 9 名，Grok 4.7 也失 12 分，显示旧指数高估了部分模型。
3新版首次将 60% 权重交给私有测试集，以防止针对公开考题的 benchmaxxing 专项训练。
4Nathan Lambert 认为部分实验室缺乏复杂 RL 基础设施也是原因；私有题目可能出错，AA-Omniscience 曾因此修订。

决策要点

评测机构转向私有考题后排名重排，开源模型此前的追赶表现有相当部分来自对公开基准的过拟合。

影响：依赖公开榜单的模型选型结论可能需要重算；私有考题让外部更难发现基准题目错误，企业验证模型能力时要更多依靠贴合自身业务的内部评测。
参考：可考虑把公开榜单分数仅作初筛，用自有业务题库对候选模型做第二轮评测；值得重新核对基于旧版指数做出的选型或采购清单。
依据：分差与排名变化来自 Artificial Analysis 官方口径，暂无独立验证；Lambert 对成因的分析为研究者个人观点；AA-Omniscience 曾因题目歧义被官方修订。

正文

Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to drop five spots in the rankings. The model originally launched in fourth place, but it falls to ninth behind eight closed models in version 5. Artificial Analysis now puts the gap between open models and the top of the leaderboard at 12 points, up from 3 points previously.

For the first time, 60 percent of the index relies on private test sets to prevent "benchmaxxing," the practice of training models specifically on known test questions. Kimi K3.1 Pro's drop suggests the old index made some models look better than they were. Grok 4.7 also lost 12 points.

AI researcher Nathan Lambert thinks benchmaxxing is only part of the story. Many training gains today happen in reinforcement learning environments with tasks that resemble the benchmarks. Keeping up across multiple domains at once takes sophisticated RL infrastructure, and not every lab has it.

Closed test sets do have a downside, since outsiders can no longer spot errors in the prompts. The AA-Omniscience knowledge test proves those errors exist. Artificial Analysis had to revise ambiguous and flawed questions in that benchmark for the new index.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.