LLM Dataset Curation and Processing

This post details advancements in LLM dataset curation and processing, introducing SlimPajama, a 627B token, extensively deduplicated, multi-corpora open-source dataset for training large language models. It details the process of cleaning and deduplicating the RedPajama dataset, removing 49.6% of bytes to improve data quality and compute efficiency. The post also releases the tools developed for this process, enabling community use for creating high-quality, deduplicated datasets at trillion-to-scale. The processing pipeline involved optimizations for distributed, multi-threaded, and memory-efficient MinHashLSH deduplication on trillion-token datasets, including rewriting parts of the datasketch library.