Scaling Laws and Interpretability of Learning from Repeated Data

This post systematically studies the effects of repeated data in LLM training. It finds a strong double descent phenomenon where repeated data can increase test loss midway through training. A predictable range of repetition frequency leads to severe performance degradation; for instance, repeating 0.1% of the data 100 times can degrade an 800M parameter model's performance to that of a 400M parameter model. The work connects these observations to mechanistic interpretability, showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization.