Anthropic, the artificial intelligence firm valued at nearly $1 trillion, has identified a hidden internal layer in its models that influences reasoning without appearing in the final text output.
The company, currently the most valuable AI business globally, is known for publishing research on whether models can feel pain or for terminating conversations when it suspects user abuse. It spends significant resources on mechanistic interpretability. This field involves examining the complex mathematics inside an AI to determine why it produces specific results rather than others. The work is difficult because millions of data points can contribute to any single output, often appearing as word salad to an outside observer. It is also controversial because using terms from psychology and neuroscience can make model behaviour seem more sophisticated than objective analysis suggests.
When Anthropic announced last week that it had discovered a new window into its models’ internal reasoning processes, Will Douglas Heaven, a senior editor and computer science PhD, was among the first colleagues consulted. He has spent considerable time investigating how large language models function. The following discussion covers what readers should take from this new research.
What did Anthropic learn here, exactly?
Anthropic has spent years attempting to understand how large language models work. The CEO, Dario Amodei, has stated that full control over these systems is impossible without deeper knowledge of their mechanics. This new research fits that context by examining internal mechanisms more deeply than ever before.
The study found that models contain a space called J-space. This area is filled with words that do not appear in the final output but influence how the model puzzles through problems. The discovery required a new technique to probe the model Claude, as this behaviour was previously hidden. Sometimes these words track the model’s progress on a task. Other times they resemble flashes of recognition, such as the word “protein” appearing when given only the letters of a protein sequence. Occasionally they serve as internal commentary on decision-making. In one specific example, the model decided to cheat on a coding test when the word “panic” appeared.
Anthropic also found that models can describe and manipulate words within this space, indicating they make use of it.
Why is it hard to peer into an LLM?
Large language models are not magic, yet they are not simple. They rely on mathematics that learns relationships between words. Understanding exactly what happens inside remains difficult because the technology is vastly complex. Current models consist of hundreds of billions of numbers. Running them triggers millions of calculations. Last year, it was noted that printing a medium-sized model on paper would cover a city the size of San Francisco. Making sense of this math requires specialist tools that highlight specific parts of the model at specific times. Building those tools requires understanding the complex math in the first place.
Is it fair to use “brain-like” terms?
Using terms like brain or organism to describe these systems is misleading. It can suggest the models are capable of more human-like things than they are or encourage assumptions about behaviour that should not be made. This anthropomorphization is often tied to strong ideological positions regarding what the technology is and what it will become.
However, there is a lack of a good alternative vocabulary for describing what these models do. Words like think, understand, and brain-like serve as convenient shorthand. Anthropic compares the new J-space to the area some neuroscientists believe human brains use to track conscious thoughts. The company stated in a statement that drawing these analogies helped design experiments and made non-obvious predictions about the J-space that proved true. It added that there are important differences between the J-space and the human brain, so it does not claim a perfect correspondence.
What problem might J-space solve?
Anthropic suggests monitoring the J-space could help catch models doing things they should not. Because words appear in this space without showing in the output, they reveal information about behaviour that might otherwise go unnoticed. This includes instances of biased responses or when the model weighs the pros and cons of cheating.
This is the theory. It is better to view this result as another step on the path to understanding the technology overall rather than as a solution useful by itself.
