Why Nobody Has Solved Continual Learning

This talk details the fundamental challenges in continual learning for AI models. It contrasts Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), highlighting SFT's ability to introduce new knowledge but its susceptibility to catastrophic forgetting and degradation of general capabilities due to off-policy learning. RL is presented as an on-policy alternative that avoids forgetting but is limited to reinforcing existing behaviors and struggles with novel knowledge integration. The talk also discusses the use of distillation for warm-starting RL, noting its initial performance benefits but eventual plateauing at lower levels compared to RL alone. The speaker emphasizes the need to tune the ratio of SFT and distillation to RL based on task requirements, differentiating between narrow tasks and general agents. The ongoing research in this area by frontier labs is acknowledged.