LLM Feature Representation and Superposition

5 posts · Mar 2022 – Oct 2023

What this trail covers

Anthropic is investigating how large language models represent features, particularly the phenomenon of 'superposition' where models can represent more features than their dimensionality. This involves using toy models (small ReLU networks on synthetic data) to understand the mechanisms and conditions under which superposition occurs, its benefits (compression beyond linear models), and its costs (interference requiring nonlinear filtering). This post extends the investigation to limited dataset sizes and explores the relationship between in-context learning and induction heads.

All Anthropic trails

Oct 2023Decomposing Language Models Into Understandable ComponentsThis post introduces the concept of 'features' as more interpretable units than individual neurons in language models. It details a method using dictionary learning to decompose a layer of a small transformer model into over 4000 features, which represent specific concepts like DNA sequences or legal language. Human evaluators and an autointerpretability approach confirm these features are significantly more interpretable than individual neurons. The post also demonstrates that activating features can predictably alter model behavior and that learned features are largely universal across models. It highlights the potential for scaling this decomposition to frontier models, with engineering challenges identified as the next hurdle.
Oct 2023Towards Monosemanticity: Decomposing Language Models With Dictionary LearningThis post introduces dictionary learning as a method to decompose transformer layers into monosemantic features, which are linear combinations of neuron activations. It demonstrates that these features can represent specific linguistic or structural properties (e.g., DNA sequences, legal language, HTTP requests) more effectively than individual neurons. The work shows that a layer with 512 neurons can be decomposed into over 4000 such features, providing a path to understanding complex neural networks.
May 2023Distributed Representations: Composition & SuperpositionThis post expands on the concept of superposition in LLMs by introducing and contrasting it with 'composition' in distributed representations. It uses examples from neuroscience to illustrate how these two ideas can be understood as separate dimensions, rather than a single spectrum. The authors propose that composition relates to how distinct features are represented by distinct neurons or groups of neurons, while superposition relates to how multiple features can be encoded within the same neuron or group of neurons. The post highlights a fundamental tension between these two concepts, suggesting a trade-off in their application.
Mar 2023Privileged Bases in the Transformer Residual StreamThis post investigates the phenomenon of privileged bases in the Transformer residual stream, which contradicts theoretical expectations of arbitrary basis directions. The authors explore potential causes, including Layer Normalization and finite-precision floating-point calculations, and confidently rule them out. They provisionally conclude that the per-dimension normalizers in the Adam optimizer are responsible for the observed basis alignment.
Jan 2023Superposition, Memorization, and Double DescentThis post extends previous work on superposition in toy models to limited datasets, investigating overfitting. It finds that overfitting in this regime corresponds to storing data points in superposition, while generalization corresponds to storing features in superposition. The authors observe a double descent phenomenon as the model transitions between these two regimes.
Advertisement
Sep 2022Toy Models of SuperpositionThis post introduces the concept of 'superposition' in ReLU networks, where models can represent more features than their dimensionality. It demonstrates this phenomenon using toy models with sparse input features, showing that superposition allows for compression beyond linear models but introduces interference that requires nonlinear filtering. The work explores the conditions under which superposition occurs and its implications for feature representation.
Jun 2022Softmax Linear UnitsIntroduced Softmax Linear Units (SoLU) as an activation function that increases the fraction of interpretable neurons in MLP layers with minimal performance impact. Studied SoLU models to gain insights into information processing in transformers and found evidence supporting the superposition hypothesis, suggesting SoLU might increase interpretability by hiding other features.
Mar 2022In-context Learning and Induction HeadsThis post investigates the relationship between in-context learning and induction heads in LLMs. It demonstrates that induction heads, a specific type of neural circuit, are responsible for in-context learning by enabling models to recall and apply patterns seen in the prompt. The research uses toy models to show how these heads emerge and function, and how they can be related to the phenomenon of superposition, where models can represent more features than their dimensionality. The post also explores the limitations and potential interference issues associated with this mechanism.