开源基准 CloudOpsBench 揭示推理是 AI 智能体云运维的最大瓶颈
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
- 1密歇根大学团队发布开源基准 CloudOpsBench 1.0，含 45 个 AWS 云运维任务，由资深 DevOps 工程师构建
- 2最强模型也无法全对，Claude Fable 5.1 解出 60%，GPT-6 Astra 解出 53%，没有模型完成全部任务
- 3团队指出主要瓶颈是推理而非知识，约 80% 失败源于误判任务范围、漏查相关服务或过早停止
- 4约 20% 的错误是错误操作，部分对基础设施不可逆；全部任务在真实 AWS 与模拟器中同标准运行
智能体做云运维卡在推理而非知识，且约两成错误可能对基础设施造成不可逆后果，直接放手让 Agent 改生产环境风险不小。
- 影响：最强模型也只解出六成任务，云运维自动化短期内仍需人工审核兜底；不可逆操作意味着 Agent 接入生产环境前要先收紧权限和审批；基准以 Apache 2.0 开源，可作为内部选型的测试工具。
- 参考：可考虑用 CloudOpsBench 的 45 个任务先测自家候选模型在 AWS 场景的真实水平；值得盘点哪些运维操作不可逆，把这类操作排除在 Agent 自动执行范围之外或要求人工确认。
- 依据：结果来自密歇根大学 Ang Chen 团队的基准测试，任务在真实 AWS 与模拟器中同标准运行，尚无第三方独立复测数据。
CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
来源：The Decoder：AI News · the-decoder.com