How we fine-tuned Llama2-70B to pass the US Medical License Exam in a week

This post details the fine-tuning of Llama-2 70B on a custom medical dataset to pass the US Medical License Exam (USMLE). It highlights the challenges of fine-tuning large models (70B parameters) on traditional GPU hardware due to memory constraints (requiring ~1TB for parameters and optimizer states), contrasting it with the ease of training on Cerebras CS-2 systems. The Cerebras platform's disaggregated memory architecture (12TB available) easily accommodates the model's memory requirements, allowing for straightforward training without complex memory management techniques like LoRA. The project was completed within a week, with the 70B model fine-tuning starting Friday and finishing Tuesday, achieving a 72% score on USMLE sample questions, outperforming Med-PaLM and ClinicalCamel-70B. The post emphasizes the simplicity and efficiency of the Cerebras AI training platform for large model fine-tuning.