Anthropic has just released a groundbreaking paper that inspects a Large Language Model from the inside for the very first time.
The Problem of Neural Networks as Black Boxes
Until now, such models acted like black boxes: input producing an output without a clear understanding of the reasons that lead to it.
More specifically, as you might know, these models are composed of neurons that are or are not activated by activation functions. The combination of neurons that get activated when a certain input is passed make up features, and the combination of features create an internal state of the model.
Well, most LLM neurons are uninterpretable, which stops us from mechanistically understanding the models.
A new technique to map the internal states of the model
The Anthropic team used a technique called "dictionary learning", borrowed from classical machine learning, which isolates patterns of neuron activations that recur across many different contexts. In turn, any internal state of the model can be represented in terms of a few active features instead of many active neurons, which means decomposing the model in features.
This technique was previously used on very tiny models and showed promising results. This is what motivated the researchers to apply the “dictionary learning” technique to Claude 3 Sonnet. And the results are mind-blowing.
The Anthropic team was able to extract millions of features which creates a conceptual map of its internal states halfway through the models computation. These features are remarkably abstract, often representing the same concept across contexts and languages, even generalizing to image inputs.
What the inside of an LLM looks like
What they found was that that the distance between similar features reflected their similarity. In other words, the model has a mapping of concepts that are as close to each other as their associated features.
For examples, looking near a feature related to the concept of "inner conflict", they find features related to relationship breakups, conflicting allegiances, logical inconsistencies, and so on.
But what’s even more interesting is that it is possible to artificially manipulate those features, and that emphasizing or deemphasizing certain features changes the behavior of the model. This is a groundbreaking result and a huge step forward for ensuring the safety of these models.
For example, researchers found a specific feature associated with blindly agreeing with the user. Artificially activating this feature completely alters the response and behavior or the model. |