Comment ranker - An ML-based classifier to improve LLM code review quality using Atlassian’s proprietary data

This post details the development and impact of the 'comment ranker,' an ML model designed to improve the quality of code reviewer agent comments. It describes the binary classification task of predicting comment resolution rate (CRR) using a fine-tuned ModernBERT model. The ground truth is derived from Atlassian's internal dogfooding data, where user code changes after a comment indicate value. The model's training, fine-tuning process, and the importance of model refresh due to data drift are explained. The post highlights the comment ranker's success in increasing CRR, its adaptability to different LLMs (GPT-4o, Claude Sonnet 3.5), and positive user feedback, positioning it as a proprietary moat for Atlassian's code review capabilities.