Beating proprietary models with a quick fine-tune

What this post added

This post details the process of fine-tuning embedding models to achieve better accuracy and lower cost than proprietary alternatives. It covers the 'why' of fine-tuning, including the data flywheel concept and the benefits of open-source models. It then delves into the 'how,' discussing dataset selection (using the Quora duplicates dataset as an example), choosing a base model, and acquiring training infrastructure. A key contribution is the description of running a grid search over hyperparameters (base model, dataset size, embedding dimensions) to optimize the fine-tuning process, leveraging Modal's autoscaling GPU capabilities for efficient experimentation.