The most consequential design choice in an AI system is often invisible by the time anyone uses it. It is not the chat interface, the model name, or the prompt template. It is the material that was allowed to teach the model in the first place: millions or trillions of examples, each carrying some small vote about what language, images, code, or decisions should look like.
Training data is the collection of examples, records, documents, images, audio, code, labels, and other inputs used during training to update an AI model’s parameters. For large language models (LLMs), it can include web pages, books, code repositories, academic text, conversations, licensed archives, instruction-response pairs, and human preference ratings. The model does not store that collection as a tidy library. It extracts statistical patterns from it, which is powerful, useful, and a little unsettling when the collection was assembled with the organizational standards of a garage after a move.
That is why training data deserves to be treated as a supply chain rather than an ingredient. The data determines what a model sees repeatedly, what it rarely sees, what it mistakes for normal, and which gaps it cannot possibly fill. Better architecture and more compute can improve a model’s ability to learn, but they cannot teach it a domain, language, or value that never made it into the material.
The Input Supply Chain
People sometimes talk about a model as though it simply “learns from the internet.” That phrase is technically convenient and operationally useless. The internet is not one dataset. It is a messy collection of sources with different authors, rights, formats, audiences, languages, quality levels, and reasons for existing. A product manual, a spam page, a scholarly article, a ten-year-old forum thread, and an autogenerated list of cities are all text. They are not equally valuable evidence about how a useful system should behave.
Data curation is the work of selecting, filtering, organizing, and documenting material before it reaches a model. It starts with sourcing, but it does not end there. Teams parse files, remove boilerplate, identify languages, check licenses, detect personally identifiable information, filter low-quality text, label material when necessary, and decide what to exclude. Each step looks mundane when viewed alone. Together, they determine the model’s education.
The Pile made this point early by assembling 22 diverse, relatively high-quality text components instead of relying on a single web crawl. Models trained on that mixture improved across the corpus’s components and on downstream evaluations (Gao et al., 2021). The important lesson is not that 22 is a magical number. It is that a corpus has a point of view. Choose mostly scientific writing and code, and the model develops one kind of fluency. Choose broad conversational web text, and it develops another. The source list is a capability plan disguised as a spreadsheet.
This also explains why representation is a technical issue, not a ceremonial sentence in a project brief. A dataset can be enormous while giving shallow coverage to particular languages, regions, occupations, writing styles, or forms of expertise. The BigScience ROOTS corpus documented a values-driven attempt to build a multilingual collection spanning 59 languages (Laurençon et al., 2022). Coverage alone does not guarantee fair outcomes, but a model cannot offer reliable behavior for people whose language or context was barely represented among its examples.
Quality Is a Choice
Raw volume is seductive because it is easy to count. A team can announce that it has acquired another trillion tokens, everyone can nod solemnly, and nobody has to decide whether those tokens are useful. But a page full of navigation links, copied disclaimers, search-engine bait, corrupted text, or recycled paragraphs still consumes compute. The model is very patient. It will learn from bad material for as long as the hardware budget permits.
Quality filtering asks a more useful question: does this example contain the kind of signal the model should spend time learning? That does not produce one universal scoring rule. A high-quality source for a coding model may look different from a high-quality source for a multilingual assistant, a medical model, or a creative-writing tool. Quality is connected to the intended job, the expected users, and the failure modes the team can tolerate.
The FineWeb project documented its filtering and deduplication choices across a large web corpus, including a more selectively filtered educational subset that improved performance on knowledge- and reasoning-oriented evaluations (Penedo et al., 2024). DataComp-LM reached a similar practical conclusion from controlled experiments: changing curation strategies can improve results even while reducing the compute needed to reach them (Li et al., 2025). More data is not automatically more education. Sometimes it is simply more homework from the kid who copied the textbook badly.
The table’s decisions are related, but they are not interchangeable. A perfectly deduplicated corpus can still be badly sourced. A licensed corpus can still be unbalanced. A wonderfully documented dataset can still be a poor fit for the model being built. Good curation is less like one quality gate and more like a series of small, explicit decisions that refuse to hide behind the word “data.”
Duplicate Text Casts Duplicate Votes
A repeated sentence is not just a storage nuisance. During training, every appearance gives the model another opportunity to adjust its parameters around the same pattern. Repetition can be intentional, such as common phrases in a language or multiple translations of a document. It can also be accidental, caused by mirrors, reposts, scraped templates, syndicated articles, and an internet that has discovered the miracle of copying and pasting.
Deduplication identifies and removes exact or near-exact repeats before training. Its benefits are more interesting than saving disk space. In a study of language-model datasets, researchers found a 61-word sentence repeated more than 60,000 times in C4. Deduplicated training data led to models that emitted memorized text far less frequently and reached equal or better accuracy with fewer training steps (Lee et al., 2022). The repeated sentence had become an extremely loud member of the class.
That does not mean a team should delete every similar-looking passage. Near-duplicate detection has thresholds, and thresholds make judgment calls. Two documents may share a template while containing distinct facts. Parallel translations can look repetitive at the structural level. Rare material may appear similar enough to a common source that an aggressive filter removes the very examples needed to improve coverage. The sensible practice is to treat deduplication as a reviewable policy, with samples, thresholds, and exceptions, rather than a button labeled “make corpus good.”
It also intersects with privacy and memorization. Training data that contains sensitive or personally identifying material can create risks that are difficult to address after training. The right response is not to promise that a model will politely forget. It is to build detection, minimization, and review into the pipeline before the data becomes part of the learning signal.
The Mixture Becomes the Curriculum
A modern LLM is rarely trained on one monolithic dataset. It learns from a data mixture, a sampling plan that determines how frequently it encounters each source category. A corpus might contain code, mathematics, books, multilingual text, documents, synthetic instruction examples, and web data. Their raw token counts are not destiny. A team can sample a smaller high-value source more often, reduce exposure to a broad but noisy source, or shift the mixture as training moves from general pretraining toward a specialized phase.
That sampling plan is a curriculum. It tells the model what to practice most. If code receives more weight, the model gets more chances to absorb programming patterns. If a language is underrepresented, the model may need more of that material than raw availability would suggest. If a source has weak quality controls, upweighting it can amplify its flaws with the efficiency of a very expensive photocopier.
Research on mixture design finds that quality, quantity, and diversity need to be balanced rather than treated as competing slogans. Held and colleagues found that simple heuristics which account for data size and diversity can be unexpectedly strong, while utility estimates can further reduce the cost of finding useful mixtures (Held et al., 2025). The key idea is straightforward: limited compute is an experience budget. Every token selected for training is a token the model does not spend learning something else.
That budget matters at the largest scales, too. Hoffmann and colleagues showed that, under a fixed compute budget, scaling training tokens alongside model size could outperform a much larger model trained on less data (Hoffmann et al., 2022). This is not a command to collect everything in sight. It is a reminder that data is a first-class design variable, not the cardboard box the real engineering arrived in.
Synthetic data belongs in this discussion as one possible source type. It can help create edge cases, expand scarce examples, or provide task-specific material. It can also repeat a model’s own mistakes, flatten useful variety, or conceal where the examples came from if the generation pipeline is poorly documented. Synthetic examples need provenance, quality checks, and mixture limits just like data collected from people or public sources. Artificial does not mean consequence-free.
The Paper Trail
A model’s behavior may depend on thousands of source decisions made months earlier by several teams and automated pipelines. Without a record of those decisions, debugging turns into a séance. Someone notices that the model handles one language poorly, repeats a familiar phrase, or fails an evaluation, and the most important question becomes impossible: what training material could have caused this?
Data provenance is the record of where training material came from, how it was collected, what transformations it underwent, what rights or restrictions apply, and which training run used it. It is sometimes described as compliance paperwork. That is underselling it. Provenance supports reproducibility, incident response, removal requests, bias analysis, and ordinary engineering questions such as “Which filtering change affected this model?”
The need is especially acute for foundation models assembled from broad web-scale sources. Researchers have documented how authenticity, consent, privacy, representation, and licensing information is often incomplete or difficult to trace through current training-data practices (Longpre et al., 2024). Publicly available does not mean clearly licensed, representative, accurate, or appropriate for every purpose. Those are different questions, and collapsing them into “it was online” is a poor substitute for an answer.
Dataset documentation makes that record usable. The idea behind datasheets for datasets is refreshingly ordinary: describe a dataset the way an electronics maker describes a component, including its purpose, composition, collection process, recommended uses, and limitations (Gebru et al., 2021). A documentation record will not fix a flawed dataset by itself, but it lets people see the flaw before they build an entire product around it.
Some current corpora are trying to make these properties visible at scale. Common Corpus documents data provenance, filtering, license information, multilingual coverage, and removal of personally identifying information for an openly licensed pretraining collection (Langlais et al., 2026). The broader practice is what matters: every important source should carry enough context for a later team to decide whether it belongs, how it was changed, and what would happen if it had to be removed.
Questions about consent, copyright, privacy, and applicable regulation depend on jurisdiction, contracts, and the particular use. They require appropriate legal and policy expertise. Training-data engineering cannot replace that expertise, but it can make meaningful review possible by keeping the sources, permissions, and transformations visible instead of turning them into a mystery before the model has produced its first token.
Keep the Final Exam Separate
A model should be tested on material it did not train on. This is basic scientific hygiene, but web-scale training makes it surprisingly hard. Public benchmarks, answer keys, textbooks, solved problems, and copies of those materials can appear in scraped sources. Once evaluation examples leak into training data, a model may score well because it has seen a familiar worksheet rather than because it can solve a new problem.
Benchmark contamination is the inclusion of evaluation material, answers, or close variants in training data. The problem is not that a language model learns history, algebra, or common knowledge. The problem is that a benchmark is supposed to measure generalization, and a leaked item changes the question from “Can it reason?” to “Does this look familiar?” The second question has its uses, but it is not an exam.
Researchers investigating contamination in modern LLM benchmarks developed methods for both open and proprietary systems, finding signs that public evaluation material can be exposed to models in ways that complicate reported results (Deng et al., 2024). Preventing this requires decontamination work: tracking held-out datasets, checking for lexical and semantic overlap, versioning exclusion lists, and treating benchmark changes as part of the training-data pipeline.
This is also where model evaluation becomes inseparable from data curation. An evaluation result is only as honest as the boundary between training material and test material. If that boundary is blurry, the benchmark can still produce a tidy number. It just cannot tell you much about what the model will do when it meets something genuinely new.
The Decisions That Remain
Training data is often described as the raw material of AI. “Raw” is the misleading word. By the time data reaches a serious model, it has been gathered, filtered, weighted, transformed, documented, and excluded according to a long list of choices. Those choices shape capability and risk before the optimizer ever starts its work.
The mature approach is not to find a perfect dataset. There is no such thing, and waiting for one is a dependable way to keep a project from learning anything at all. It is to make the tradeoffs inspectable: know what sources are included, why they were selected, who is represented, what has been filtered out, what rights apply, how duplicates are handled, and whether the final exam stayed outside the textbook.
Models will become more capable, training runs will become more expensive, and datasets will become harder to source cleanly. That only makes the underlying lesson more useful. The data pipeline is not the prelude to model development. It is model development, wearing work gloves.
