Constitutional AI: Harmlessness from AI Feedback

This post introduces the concept and initial implementation of Constitutional AI for training harmless AI assistants. It details a two-phase process: a supervised learning phase where an initial model generates self-critiques and revisions, followed by fine-tuning on these revisions; and a reinforcement learning phase using AI-generated preferences (RLAIF) to train a reward model and subsequently fine-tune the AI. The post highlights the use of a list of principles (the 'constitution') as the sole human oversight and demonstrates the ability to train a non-evasive, harmless assistant that explains its objections to harmful queries. It also notes the benefit of chain-of-thought reasoning in both phases for improved performance and transparency.