Direct Preference Optimization for LLMs

This post details the application of Direct Preference Optimization (DPO) for fine-tuning Large Language Models (LLMs) to align with human preferences. It introduces BTLM-Chat, an enhanced version of BTLM fine-tuned using DPO, available on Hugging Face. The post explores the advantages of DPO over traditional Reinforcement Learning from Human Feedback (RLHF), highlighting its simpler training method and improved stability. Experimental results show that DPO leads to noticeable improvements in downstream task performance, particularly for conversational datasets, and that beta selection is key to DPO performance. The post also investigates the impact of precision settings (fp32 vs. bfloat16) and LoRA on DPO efficacy, and examines the extent of original information retention during DPO training.