Superposition, Memorization, and Double Descent

This post extends previous work on superposition in toy models to limited datasets, investigating overfitting. It finds that overfitting in this regime corresponds to storing data points in superposition, while generalization corresponds to storing features in superposition. The authors observe a double descent phenomenon as the model transitions between these two regimes.