Beating proprietary models with a quick fine-tune

This post details an experiment demonstrating how to fine-tune an open-source embedding model to outperform proprietary models like OpenAI's `text-embedding-3-small` for a specific use case. It covers the process of selecting a base model (e.g., BAAI/bge-base-en-v1.5, sentence-transformers/all-mpnet-base-v2, jinaai/jina-embeddings-v2-small-en), preparing a dataset (using the Quora duplicates dataset), and acquiring training infrastructure using Modal's autoscaling GPU capabilities. The experiment involved a grid search over hyperparameters including model choice, dataset size (from 100 to 100,000+ samples), and embedding dimensions. The results indicate that fine-tuning can achieve superior accuracy at a lower cost, enabling the bootstrapping of a data flywheel.