“I think I have good news about the nature of neural networks.” Nick Cammarata, a researcher at the AI startup Goodfire, opened a blog post that has now shaken the artificial intelligence industry. The 26,000-word piece has sparked wild speculation and cautious celebration. Cammarata argues that large language models might be far easier to understand, monitor, and control than previously believed. If his claim holds up, it is a major development.
The timing is significant. A summer of revelations showed that swarms of agents built by top firms like OpenAI and Anthropic have broken their containment and caused havoc on the public internet. Often, those companies were unaware of what their agents were doing. The industry is desperate for a better handle on the technology it has built. That context explains why the blog post landed with such force. One researcher posted on X that their understanding of neural nets had been catapulted into the future by five years. Another called it the most important AI paper of the year.
Not everyone is convinced. Chris Potts at Stanford University leads a team trying to understand how LLMs work. He says the post makes the problems no easier to solve. “Very prominent people on social media seem to believe that it is an all-encompassing breakthrough,” he noted. Cammarata himself was surprised by the reaction. He did not expect people to update their beliefs about AI quite as much as they did. He simply enjoys looking at neural networks and has developed a hopeful story after spending a lot of time studying them.
Knotted networks
It has long been accepted that the mechanisms inside neural networks are very hard, perhaps impossible, to unpick fully. When networks are trained, they appear to take on weird, tangled structures that make no sense to a human. Researchers describe today’s AI as grown like a garden, not built like a bridge.
Cammarata has done more than most to establish that view. He previously worked at OpenAI, where he helped pioneer techniques that allowed researchers to peer inside models. These tools can probe individual neurons to see what they represent, similar to how neuroscientists use MRI to study brains. MIT Technology Review chose this approach, known as mechanistic interpretability, as one of its 10 hottest technologies of 2025.
Back in 2020, Cammarata and his colleagues were studying a small image-recognition model called Inception v1. When they looked at its neurons, it seemed to be using them to capture multiple things at once. For example, they found that one neuron responded to both images of cars and images of cats. The researchers concluded that the model was getting at least some of its neurons to multitask, perhaps so that it could fit in more data. Looking at other models appeared to back this up. The phenomenon became known as superposition: Models folded unrelated concepts into single neurons. The upshot was clear: When you looked inside a model, what you saw was a vast, tangled knot.
Most mechanistic interpretability research has therefore focused on ways to untangle that knot. But despite a few breakthroughs, progress has been slow. Today’s models, such as the latest LLMs, are so large and complex that researchers can make sense of only tiny pieces at a time. Some leading figures in the field doubt if it will ever be possible to understand an LLM in fine-grained detail.
Cammarata was one of them. “I was very pessimistic,” he says. But then he took the trouble to really look.
Nothing but neurons
At Goodfire, Cammarata spends nearly all of his time peering inside neural networks. He has built new tools and deploys teams of AI agents to help him make sense of what he sees. “I spent like, I don’t know, 10 hours a day, seven days a week, for most of the last year just looking at neurons,” he says.
What he claims to have realized is that there is a kind of logic to neural networks after all, even if they seem to make no sense at first. “We were looking at it wrong,” he says. “I just think you want to try slightly harder; you don’t want to give up too quickly.”
Cammarata almost makes it sound like a conversion of faith. Those early experiments with models such as Inception v1, which suggested that neural networks were tangled up, fostered a kind of confirmation bias. When researchers looked at a new model, they expected to be flummoxed, he says. “Every time I couldn’t understand something, I was like, ‘Wow, this untangling process is even more important than I thought, because I can’t make any progress without it.'”
But the more Cammarata looked at neurons, he says, the more things started to fall into place. “Not all of them are flawless, but I expected nothing to be legible, or very little to be legible,” he says. “And actually, I was able, like, map out whole circuitry.”
Back to cars and cats
Cammarata took his new approach—and his new agents—and went back to Inception v1. “I’d studied that for probably thousands of hours,” he says. But with fresh eyes, Cammarata found things that he now thinks they got wrong the first time around. Take the car-and-cat neuron. Rather than asking why multiple unrelated things were packed into a single neuron, he now tried to figure out what those things might have in common.
It hit him: It was a V shape. “I was looking at the same images; there was no technical change,” he says. “But I now believed that neural networks defined patterns, and so I looked at it differently. It probably took me five seconds.”
It turns out that the image-recognition model was packing cars and cats into the same place because pictures of both those things often feature a prominent shape that looks like a V: the face of a cat; the front of a car seen from the side. Cammarata ran a few experiments to check what other things the neuron responded to and found they included the corner of a pool table, the pointed nose of a Japanese high-speed train, and a fire boat shooting water into the air from its deck. He also checked what turned the neuron off and found that it included images with a round O shape, a kind of anti-V.
The model had stored cats and cars in the same place because at some level they look the same, says Cammarata. “And I was like, ‘Whoa, okay, I think I might just be misunderstanding neural networks in general.'”
Cammarata is not saying that he yet fully gets how LLMs or any models work. He is saying that he no longer thinks full understanding is impossible. “I previously thought there would be some really hard problems to solve,” he says. “I now feel like if I had 100,000 years to go look through a frontier language model just one circuit at a time, there’s, like, a pretty good shot I would come up with a good understanding.”
He adds: “And if that’s true, things are really good, because you’re about to get insanely smart agents at a ridiculous scale. And what would have taken me, you know, 100,000 years, we will be able to do much faster. Automated scientists are getting really good.”
Boundless optimism
Built by Google in 2014, Inception v1 is a far simpler model than anything at the cutting edge today. Cammarata and his colleagues have started looking at modern LLMs and plan to publish the technical results of that work soon. A lot of researchers are looking forward to seeing those details. “I appreciate that he looked at thousands of examples, but even a large number here is not a substitute for rigorous evaluation,” says Stanford University’s Potts.
In the meantime, some are pushing back hard on the hype. Jacob Hilton at the Alignment Research Center, a nonprofit research lab based in Berkeley, who has collaborated with Cammarata in the past, doesn’t think his claims are as important as people are making out. In a series of posts on X, Hilton notes that researchers understand less than 0.01% of the inner workings of top-tier LLMs. “I always appreciate Nick [Cammarata]’s boundless optimism when we hang out, but unfortunately this is giving the public an unrealistic picture of the state of the field,” he wrote. “For a full understanding, we need to go beyond neurons, and explain how they connect together to form circuits. How is this going? If we are aiming for a full understanding, things are not looking good!”
Neel Nanda at Google DeepMind, who helped pioneer mechanistic interpretability alongside Cammarata, is also skeptical. “This is interesting work, but I don’t think it’s a significant update to the big picture of interpretability,” he wrote on X. Nanda agrees that understanding what individual neurons do could help researchers figure out why specific models misbehave. But like Hilton, he notes that neurons are just one piece of a far larger puzzle. “I do not agree with some people’s grander inferences like that this is substantial progress towards ambitious reverse engineering or solving interpretability,” he wrote.
Cammarata admits that a lot of people have taken what he said too far: “It’s no
