Publication
KletterMix: Climbing Toward High-Quality German Pretraining Data
Maurice Kraus; Ruben Härle; Sebastian Sztwiertnia; Abbas Goher Khan; Mehdi Vali; Michael Fromm; Kristian Kersting
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2606.03773, Pages 1-29, arXiv, 2026.
Abstract
High-quality pretraining data is a central ingredient in modern language models,
but German-language resources remain far less developed than their English coun-
terparts: they are often smaller, less carefully curated, weakly documented, and
rarely validated through controlled training experiments. We introduce Kletter-
Mix, a high-quality German corpus for language model pretraining and annealing,
designed as a reusable dataset artifact for the natural language processing and
modeling community. KletterMix is built by translating a state-of-the-art English
pretraining corpus into German while preserving document boundaries, metadata,
source structure, and topical diversity. This construction yields a German corpus
with the scale and diversity of a modern pretraining dataset, while enabling direct
comparison to its English source. We document the dataset through a broad set of
corpus-level analyses, including translation quality, document length distributions,
topic coverage, source composition, and geographic metadata. Using COMETKiwi,
we show that the translated documents achieve strong quality across diverse do-
mains, suggesting that careful translation can preserve much of the semantic and
stylistic richness of the original corpus. Beyond dataset construction, we evaluate
KletterMix as training data. Through controlled pretraining and annealing ablations
against established German corpora, we show that models trained on KletterMix
achieve measurable improvements on German-language downstream evaluations.
These results demonstrate that carefully curated translated data can substantially
strengthen the German pretraining data ecosystem.