A General Language Assistant as a Laboratory for Alignment

This post introduces foundational techniques and evaluations for aligning a general-purpose language assistant with human values (helpful, honest, harmless). It demonstrates that simple interventions like prompting, when scaled with model size, improve alignment without degrading performance. The work compares three training objectives: imitation learning, binary discrimination, and ranked preference modeling, finding ranked preference modeling to be the most effective and scalable. It also proposes 'preference model pre-training' as a method to enhance sample efficiency during fine-tuning on human preferences.