Mitigating memorization in LLMs

This post explores mitigating memorization in LLMs by investigating methods like divergence decoding, distillation, prompt engineering, and natural language autoencoders. It details experiments on a cricket match dataset to measure and reduce reliance on memorized pretraining knowledge, with divergence decoding showing promise in reducing pre-cutoff performance without harming post-cutoff performance.