Data Prep Kit: Open Source Data Engineering for LLMs

Introducing Data Prep Kit (DPK) - an open source data engineering framework for LLMs. DPK was developed internally by IBM to assist with the development of its open source Granite family of LLMs and released in 2024. DPK is built on Kubeflow pipelines, running on scalable compute ranging from a developer's laptop to massive clusters. In addition Kubeflow pipelines allows the community to collaborate on common LLM data engineering workflows problems, such as determining licensing state of the data and determining the state of its GDPR compliance.