Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

This post introduces the application of preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models for helpfulness and harmlessness. It details how this alignment training improves NLP evaluation performance and is compatible with specialized skill training (e.g., Python coding, summarization). The authors explore an iterated online training mode with weekly updates of preference models and RL policies using fresh human feedback. Additionally, they investigate RLHF robustness, finding a linear relationship between RL reward and the square root of KL divergence.