SlimPajama: A 627B token, cleaned and deduplicated version of RedPajam

This post details the creation of SlimPajama, a 627B token dataset derived from RedPajama by extensive cleaning and deduplication. It introduces a novel, scalable infrastructure for MinHashLSH deduplication on trillion-token datasets, implemented using producer-consumer patterns and optimized datasketch logic for distributed and memory-efficient processing. The post also describes the data cleaning steps, including filtering short, low-quality documents and applying MinHashLSH with a Jaccard similarity threshold of 0.8 on 13-grams. The resulting dataset and the data processing library are open-sourced.