Cammarata发文挑战叠加理论，AI可解释性能否突破引发研究者分歧

Mechanistic interpretability tries to do more than watch what an AI model outputs. It aims to trace how the model arrives at its answers internally. Ideally, each of the millions of artificial neurons would stand for a recognizable concept. In practice, many neurons seemed to fire on completely unrelated things, which researchers call polysemanticity. One early and famous example is a neuron in the image model InceptionV1 that responded to both cat faces and the fronts of cars. Cammarata used it as an example in his own early research.

The standard explanation was superposition. The theory goes that when a network runs out of neurons, it packs unrelated concepts into the same neurons because they rarely get in each other's way. The result looks like a knot. That's why many labs train a second model, such as a sparse autoencoder, to pull the concepts apart. They then study that model instead of the original.

Cammarata played a major role in shaping this view. With Chris Olah, he co-wrote the paper "Zoom In", which introduced the first superposition circuit. In 2023, he started an automated interpretability project at OpenAI. At the time, only about 1.7 percent of neurons counted as understandable, even by the most generous measure.

The evidence may have been misread

Cammarata now thinks the most famous examples were misinterpreted. He says the cat-car neuron is really a V-shape detector. Pointed shapes like fox snouts or flames activate it, while round shapes like the letter O push its activation negative. So it doesn't represent two completely unrelated concepts like cat faces and cars, which means it isn't an example of superposition.

He also reinterprets the superposition circuit from "Zoom In," where a car seemed to hide among dog neurons. In his view, it's ordinary generalization, and the circuit simply groups cars and dogs by which way they face the camera. He blames the mess in InceptionV1's late layers on an outdated architecture and weak analysis tools.

Cammarata explains the remaining ambiguity with a pattern he calls "recursive generalization." He says a neuron grows like a branch that sprouts side shoots. One neuron that recognized the letter sequence "iy" later responded to "diy" and eventually to "do it yourself," which doesn't contain "iy" at all. Unlike superposition, these groupings can teach you something about the network.

For language models, he offers preliminary numbers. In Qwen3-32B, about 90 percent of neuron poles have one main job and about 7 percent have two related ones. Only 2 to 4 percent remain unexplained. Goodfire says it sees similar results in models with up to trillions of parameters. AI agents that analyze neurons in bulk did most of the work. Cammarata borrows from Rich Sutton's "Bitter Lesson" and concludes that in interpretability, too, simple investigation at massive scale beats clever decomposition methods.

Cammarata doesn't introduce a fundamentally new method. He goes back to the field's original approach of studying neurons directly, just with better tools, larger models, and automated agents.

Readable networks would be a big win for AI safety

If Cammarata is right, that would be good news, especially for AI safety. Right now, apart from monitoring chains of thought that keep getting less reliable, there are few ways to tell whether a model is deceiving people or behaving dangerously. Cammarata doubts that alignment, the work of steering AI toward human goals, can be solved without interpretability, because otherwise we don't know what we're building. Deception has no reason to hide from the model's own circuits, he writes. If humans become just another "reader" of the network, they can see the deception too, and errors might be fixed in a targeted way.

Roon, an OpenAI employee who posts under that pseudonym on X and is an influential voice in AI circles, was thrilled. He says he's close friends with Cammarata and would never have ended up at OpenAI without him. The work "may be remembered as a major moment in scientific history," Roon wrote. He added that the news is so good even Eliezer Yudkowsky, one of the most prominent voices warning of an AI catastrophe, will have to lower his estimated odds of doom. In Roon's view, alignment could become a "normal science."

Even Roon has some reservations. Others in the field still need to confirm the results and he also suspects that interpretability breakthroughs will push AI capabilities forward. The fact that they're coming from outside doesn't bode well for the big labs, OpenAI included.

Nanda sees a useful tool, not a breakthrough

Neel Nanda is far more measured. He's considered one of the founding figures of mechanistic interpretability and leads the interpretability team at Google DeepMind. Before that, he worked at Anthropic and as an independent researcher. His views have shifted, according to him: Since mid-2024, he's grown more pessimistic about ambitious interpretability and has leaned toward pragmatic approaches. His team deprioritized research on sparse autoencoders. They didn't think the technique was useless, but it didn't seem worth the effort the field was putting into it.

Cammarata's work is interesting but doesn't really change the big picture, Nanda writes. Neuron-based analysis could help with practical questions, like why a model did something questionable, and it's much easier to apply to frontier models. But he doesn't buy the idea that interpretability is close to being fully cracked. He expects neuron-based analysis to have serious flaws and miss a lot as well.

Nanda still thinks superposition is real. He argues that language models store far more facts than they have neurons, and weak activations in particular likely hold much of that information. He doubts that findings from small image models carry over. He also thinks the essay defines monosemanticity too loosely but does give Cammarata credit for checking his old work with modern tools.

Part of the dispute comes down to definitions: Cammarata has no problem with models representing more concepts than they have dimensions, and Nanda sees him as agreeing on that point. What Cammarata disputes is that networks arbitrarily pack completely unrelated things together.

The technical companion paper hasn't been released yet. As an interpretability company, Goodfire also has an obvious stake in the thesis. If Cammarata's reading holds up, he thinks automated researchers could do in months what took humans nearly a decade with image models. He writes that there may not be much time left for that.