Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Introduced dictionary learning as a method to decompose transformer layers into monosemantic features. Demonstrated that these features, which are linear combinations of neuron activations, can represent distinct concepts (e.g., DNA sequences, legal language, HTTP requests) and are often invisible when examining individual neuron activations. This work provides machinery to find these features in small transformer models, offering a path to understanding complex neural networks by breaking them down into interpretable parts.