ARTIFICIAL INTELLIGENCE
FineWeb2 and the problem of deciding what counts as good multilingual data
FineWeb2 and its model-filtered successor reveal why multilingual training needs language-aware cleaning, representative evaluation and transparent sampling choices.
Research checked
Newest cited publication
CERN’s server room in 2009 illustrates computing infrastructure; it does not depict FineWeb’s collection or processing facilities.
Florian Hirzinger — www.fh-ap.com · Image source · CC BY-SA 3.0
A cleaning rule that improves English training data can damage another language. A short paragraph may be perfectly useful in a language with scarce digital resources, while unusual punctuation may be normal writing rather than corruption. FineWeb2 confronts this problem at web scale. Its main paper was submitted on June 26, 2025, and a related model-based selection paper received a February 19, 2026 revision. Together they show a progression from collecting multilingual text toward testing which selection decisions improve learning. The central question is not how many languages fit into a dataset card. It is whether the pipeline preserves the right evidence for models to learn those languages well.
Counting languages does not measure usable coverage
The FineWeb2 paper describes a roughly 20-terabyte corpus with about five billion documents, created from almost 100 Common Crawl snapshots and covering more than 1,000 languages. The dataset card gives a more granular view through language-and-script pairs and their document counts. That distinction prevents a misleading conclusion: a long list of supported labels is not evidence that each language receives comparable training material. Some subsets are large enough for extensive experiments; others contain only a thin record of the language's presence on the crawled web.
Consider a hypothetical corpus with one million documents in one language and 1,000 in another. Uniform sampling over documents gives the second language only about one document per thousand samples. Uniform sampling over languages would repeat its small corpus far more often. Neither choice is automatically correct. The first risks neglect; the second risks overfitting and excessive repetition. A training recipe must disclose the sampling distribution and distinguish the volume stored on disk from the volume actually presented during learning. Dataset size and training exposure answer different questions.
Why a universal cleaning threshold can be unfair
FineWeb2 adapts filtering decisions to languages rather than applying an English recipe unchanged. Its authors tested design choices using nine diverse languages and selected evaluation tasks intended to give informative signals. The dataset documentation also exposes removed subsets, allowing others to revisit filtering choices. This matters because clean is an operational label: a pipeline decides whether a document passes specific tests. It does not directly observe a document's cultural value, factual correctness or usefulness for every possible downstream task.
An independently designed failure example illustrates the issue. Suppose a filter rejects pages with a high proportion of repeated short strings. It may successfully remove navigation menus, yet also reject a dictionary, a grammar table or verse with deliberate repetition. A length threshold might remove spam while discarding useful notices in a low-resource language. To assess such rules, sample both accepted and rejected pages and ask fluent reviewers to classify them. Reporting only examples that survive makes a pipeline look cleaner without revealing what knowledge it has removed.
Duplicates are both a nuisance and a signal
The dataset card describes global deduplication within languages, rather than treating every crawl snapshot independently. The paper also studies rebalancing based on duplication counts and estimated quality. These are related but separate operations. Removing repeated copies changes what is stored; weighting examples changes how often surviving material is learned. Neither should be confused with deleting every recurring idea. A copied page, a quotation in a new context and two independently written explanations can require different treatment even when their text overlaps substantially.
For a simple original example, imagine a page copied onto 20 mirrors and another available at one address. Sampling raw pages can give the mirrored document 20 times the exposure despite no corresponding increase in information. Deduplication can reduce that distortion. Yet aggressively retaining only unique text might remove common terminology that learners need. The experimental question is whether a particular policy improves performance on independent tasks at a fixed token budget. A cleaner-looking corpus is useful only if its changed distribution serves the stated learning goal.
The 2026 research record adds a learned selector
The related paper, Enhancing Multilingual LLM Pretraining with Model-Based Data Selection, first appeared on February 14, 2025 and was revised on February 19, 2026. It studies classifiers designed to identify structured, knowledge-rich material across languages. The associated FineWeb2-HQ dataset card describes retaining the highest-scoring ten percent of documents in each of 20 languages. Its reported efficiency comparison uses specific one-billion-parameter training experiments and named evaluation tasks. The claimed reduction in required tokens should be understood within those conditions, not as a universal sixfold reduction in every model's training cost.
A learned quality score introduces another layer of assumptions. A classifier needs examples or representations that embody its target notion of good data. It may recognize explanatory writing well while underestimating conversation, oral-history transcription or a specialist register absent from its training. My interpretation is that a selector should be treated as a model requiring evaluation, rather than a neutral meter placed over the corpus. Keeping the original data and publishing selection methods allows the community to propose alternative notions of usefulness and test their effects.
Native-language judgment belongs in the loop
FineWeb2's community annotation effort asks speakers to judge educational quality in their languages. Its published account describes the practical aim of collecting local judgments instead of relying exclusively on English-derived heuristics. Such participation can expose mistakes a generic classifier misses. It does not guarantee representativeness: volunteer annotators may cluster by region, education or internet access. The evaluation record should describe who contributed, how disagreements were handled and whether the guidelines fit the genres that matter to the intended users.
A useful follow-up would separate at least three questions. Is the text correctly identified as the stated language? Is it readable and structurally intact? Is it useful for a declared training objective? One score cannot reliably stand in for all three. A fluent speaker may recognize a well-written political argument while disagreeing with its claims; a classifier should not silently equate disagreement with poor language quality. Review sets should include informal writing, mixed-language passages and minority variants so that higher aggregate scores do not conceal predictable exclusions.
What would make the next result convincing
The dataset package identifies an Open Data Commons Attribution license, while the underlying web documents can carry their own rights and restrictions. A publicly downloadable corpus should not be described as making every embedded work freely reusable for every purpose. Its provenance fields also remain important for investigating mistaken inclusion or sensitive information. These distinctions are part of data engineering: a training artifact is easier to audit when a document retains a traceable origin and when the processing history identifies which transformations were applied.
My proposed experiment would compare several selectors at the same training-token budget, publish per-language results and retain a fluent-human audit of rejected material. It would also test a held-out genre absent from selector development. A result that improves school-style questions while degrading everyday conversation should be reported as that tradeoff, rather than compressed into a general claim of better multilingual intelligence. FineWeb2 and its successors make these questions accessible through shared data and code. Their most durable contribution may be the ability to replace assumptions about language quality with experiments that communities can inspect and contest.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every LanguageGuilherme Penedo and collaborators / arXivPublished Checked
- FineWeb2Hugging Face FineDataChecked
- Enhancing Multilingual LLM Pretraining with Model-Based Data SelectionBettina Messmer, Vinko Sabolčec and Martin Jaggi / arXivPublished Checked
- FineWeb2-HQEPFL Machine Learning and Optimization Laboratory / Hugging FaceChecked
- FineWeb2-C: Help Build Better Language Models in Your LanguageHugging FaceChecked
FOLLOW THE QUESTION
Study engineering foundations