Data Prep Kit for LLM Workloads

5 posts · Apr 2020 – Jul 2026

What this trail covers

The Data Prep Kit (DPK) is an open-source toolkit with 20+ modules for pre-processing data for code and language. It supports ingestion, document annotation, filtering, and redaction of private information, enabling developers to build end-to-end data pipelines from ingestion to tokenization. DPK modules can scale from laptop to datacenter and integrate with frameworks like Ray and Spark, offering APIs for Python, Ray, Spark, and Kubeflow Pipelines. It has been used to produce pre-training data. This post highlights the integration of DPK with watsonx.data for unstructured data processing and the creation of scalable, repeatable, and maintainable data pipelines.

All IBM trails

Jul 2026Unleash the potential of LLMs through the Data Prep KitThis post introduces the Data Prep Kit (DPK) as an open-source toolkit with over 20 modules for pre-processing data for code and language, supporting ingestion, annotation, filtering, and redaction. It emphasizes DPK's ability to scale from laptop to datacenter and integrate with frameworks like Ray and Spark, providing APIs for Python, Ray, Spark, and Kubeflow Pipelines. The post also mentions DPK's use in producing pre-training data for Granite open models and its role in enabling RAG applications by processing new PDF/HTML files for deduplication and chunking before saving to vector databases.
May 2024Use generative AI to improve document analysis and contract managementThis post details the application of watsonx.ai for contract management, focusing on automating document analysis. It describes how watsonx.ai uses NLP and machine learning to extract and classify contract clauses (e.g., payment terms, delivery terms, warranties, indemnification), compare them against guidance, and generate recommendations for alignment. Specific techniques mentioned include semantic analysis for clause extraction beyond traditional methods, and automated redlining. Example scenarios illustrate the extraction and classification of specific clause types and the generation of alignment recommendations.
Apr 2020IBM announces Elyra AI Toolkit, a set of AI-centric extensions to Jupyter NotebooksIntroduces Elyra, a set of AI-centric extensions for Jupyter Notebooks, including a visual editor for building Notebook-based AI pipelines that can be converted into batch jobs or workflows. It provides hybrid runtime support leveraging Jupyter Enterprise Gateway for distributed clusters (Spark, Kubernetes, OpenShift), enabling interactive use of cloud resources like GPUs/TPUs. Elyra also supports Python script execution and integrates Git for notebook versioning and collaboration. A shared configuration service simplifies workspace management. The pipeline visual editor is based on the Elyra Canvas component, originally from IBM Watson Studio.
Advertisement