Training a coding model to paint watercolours with TRL and OpenEnv

This post details the implementation of a Reinforcement Learning (RL) pipeline for training a coding model to generate watercolour paintings using TRL and OpenEnv. It describes the custom RL environment, which includes the p5.brush library for painting, a system prompt restricting the model to 10 specific methods, a headless Chromium renderer, and a gate to prevent cheating. The reward function is a weighted combination of a 'gate' score, a 'length' score, a pairwise judge comparing the generated image against references, and a score from the HPSv3 preference model. Three reward mixes were trained and compared: 'judge-led', 'hps-led', and 'hps-only'. The post also provides the full pipeline running on Hugging Face infrastructure, including training on Jobs, environments and scorer models on Spaces, and pairwise judging through Inference Providers.