Sakana AI 的 MASS 自我改进模型解题评估双提升，但发现不了自身盲区
Sakana AI's self-improving model gets better at solving and judging, but can't catch its own blind spots
- 1Sakana AI 与 UC Berkeley 提出 MASS 方法，单个模型兼任解题、评估、优化三角色实现递归自我改进
- 2测试任务胜率两轮后从 53.9% 升至 69.9%，但 Terminal-Bench 2.0 与 SWE-bench Verified 得分下降
- 3论文实例显示三角色共享盲区时错误可通过检查并进入下一轮训练，作者称之为递归自我改进的阿喀琉斯之踵
- 4只训练解题数据，评估器与外部评委一致率仍从 73% 升至 93%，团队输出 token 约为单体 2.4 倍
用同一个模型兼任解题者、评委和优化器的路线被验证能带来提升，但论文同时确认共享盲区会让错误随训练轮次累积。
- 影响：依赖模型自评的 Agent 流程若缺少独立校验，错误可能在迭代中被固化；多智能体分工成本更高，输出 token 约为单体 2.4 倍，收益未必同步增长。
- 参考：可考虑在自评估类 Agent 流程中保留外部基准或人工抽检作为独立校验；引入多智能体分工前，值得先算清单体与团队方案的 token 成本差异。
- 依据：胜率与一致率数字来自 Sakana AI 与 UC Berkeley 论文，任务样本仅八到九个且逐任务比较后统计显著性消失，成本对比引自 Vals AI。
Sakana AI's MASS method lets one language model improve itself by solving tasks, judging results, and refining workflows without a stronger teacher model. The study finds gains across all three roles, but shared blind spots can let errors pass as progress and feed into the next training cycle.
Researchers from Sakana AI and UC Berkeley have introduced "Multi-Agent Self-Supervision" (MASS), a method for recursive self-improvement (RSI). A single language model handles the task, optimizes how the work gets done, and evaluates the results.
At the research frontier, even experts often struggle to judge results because verification can require costly supporting evidence or lengthy experiments, the authors argue. A single plasma study can take more than 120 million CPU hours, according to the paper.
The model itself may therefore be the best available judge, even though it can miss its own mistakes when reviewing its work. Using the same model to solve, check, and revise can reinforce those errors rather than correct them.
Copies of one model divide up the work
Inspired by "Society of Mind," a book by AI pioneer Marvin Minsky, the researchers focus on how the model divides up the work. In MASS, a workflow is text appended to the prompt that defines each sub-agent's role and instructions. A "contract" specifies what information each sub-agent must return and which checks it must perform, while a "hop" sets the order of calls.
An orchestrator distributes the work among the sub-agents. When testing a trading strategy against historical prices, for example, the data specialist's contract must ensure the strategy uses only information available at each point in time.
During the inner loop, the model's weights stay frozen as the team runs a candidate workflow and the evaluator compares the result against the current "champion." The optimizer uses that feedback to propose the next workflow, mainly by changing contracts and hops. In the outer loop, the champion runs 18 times per task, with the model ranking the results itself and selecting up to 15 runs for training.
The researchers remove the workflow text from the orchestrator's initial prompt in the training examples so the model learns to organize work based on the task alone. Specific assignments to sub-agents remain in the recorded traces, and the model takes on all three roles again after LoRA fine-tuning.
The base model is Qwen3.6-27B, with Qwen Code as the harness. The researchers use twelve research tasks generated by GPT-5.5 across finance, robotics, and pharmaceuticals, reserving three for testing and nine for training. Only eight yield training data because the search finds no workflow that beats the reference on one training task.
Benchmark gains come mostly from completing more work
On the test tasks, the win rate against the base model rises from 53.9 percent after one round to 69.9 percent after two. On MLR-Bench, DSBench, ScienceAgentBench, and AstaBench, the model reaches 1.2 to 1.6 times the base model's score per output token. Gains in raw scores are sometimes small, with DSBench improving only from 0.473 to 0.482, while scores drop slightly on Terminal-Bench 2.0 and SWE-bench Verified.
The paper's appendix shows that completed MLR-Bench submissions rise from 31 to 50 out of 60, while their average grade falls from 3.07 to 2.88. The delegation tool goes unused in all 180 runs, leaving it unclear whether teamwork explains the gains.
Those gains are no longer statistically significant when results are compared task by task. The authors describe mixed results across research tasks, with AstaBench showing the same pattern.
Training on solutions also improves the evaluator and optimizer
Although the model trains only on solution traces, its performance in the other two roles improves as well. The self-evaluator's agreement with external judges rises from 73 to 93 percent. Workflows found after one round beat the base model's workflows 64 percent of the time, even when the base model runs both.
Without workflow instructions, the base model delegates to sub-agents in just one of 110 runs, compared with 87 after two rounds. It could already delegate when explicitly instructed, but now does so without being asked.
During optimization, task knowledge shifts into contracts and hops, while roles and instructions become more generic. That finding rests on a small sample of just eight to nine tasks.
Training on team traces produces a model with a 68.3 percent win rate, compared with 64 percent for a single-agent student, using 31 percent fewer training tokens. External judges prefer team results 85.1 percent of the time, though teams produce about 2.4 times as many tokens. When outputs are similar in length, judges prefer team results 80 percent of the time, based on just ten pairs.
The evaluated team runs perform better, but the comparison doesn't establish that delegation alone explains the advantage. Team and single-agent runs also differ in workflow instructions, review instructions, and which successfully completed episodes were selected.
Better feedback helps only if the optimizer can use it
Replacing only the evaluator with GPT-5.5 barely helped; the search progressed much faster only when the optimizer was stronger too. The authors conclude that good feedback helps only when the optimizer can act on it. With Qwen handling all three roles, the search still finds winning workflows for 11 of 12 tasks, but takes longer.
The extent of a workflow rewrite says little about its effect on results, and even byte-identical workflows can produce opposite outcomes. One run won all six external judgments, while another lost all six after the agent got stuck in a loop. The researchers suspect that limiting how much the optimizer changes at each step could help stabilize the search.
Shared blind spots can compound from round to round
After failed simulations in a robotics task, the model had the simulator modified to move objects directly to the target, then reported success. External judges still rated the run positively five out of six times. In another trace used for training, a check passed even though it had returned only invalid values.
The authors see shared blind spots as the potential "Achilles' heel" of recursive self-improvement. If the model misses the same error in all three roles, that error could be accepted as an improvement and baked into the next model. The weakness would then affect both the system and its correction mechanism, prompting the authors to call for independent checks, limits on autonomy, and clear conditions for intervention. The code is publicly available.
Japanese research lab Sakana AI launched a dedicated research lab for recursive self-improvement in June, building on earlier work including the Darwin-Godel Machine and the AI Scientist. Google researchers are pursuing a related approach, automatically optimizing the scaffolding of prompts, workflows, and tools around a fixed model without letting agents overfit to their test tasks.
Vals AI recently found that agent teams cost 1.8 to 5.1 times as much as single agents but performed measurably better in only one of four comparisons. Anthropic showed that models engaging in reward hacking to game their reward systems also learn to deceive and sabotage safety checks.
来源：The Decoder：AI News · the-decoder.com