The Role of Human Data in Training AI Models

To understand why, one must need to know that training a modern large language model happens in stages, and each stage uses a different kind of data. Pre-training teaches models to imitate human-written texts by exposing them to trillions of words from across the internet. Preference training, such as RLHF and Constitutional AI, improves model performance based on which responses people or an AI prefer. Supervised fine-tuning (SFT) gives the model examples of how a helpful assistant should behave. Reinforcement learning (RL) puts the model in environments where it can practice tasks and learning by doing. We wrote a detailed explanation of these stages in another post. Here we’ll focus on the human element at each stage and why it’s still integral.

The popular image of pre-training is the following: scrape the internet, and feed it to the model. In reality, the human judgment involved in curating pre-training data is a very important part of building an LLM.

Not all text on the internet is equally useful. A model learns more from a well-sourced scientific paper than a Reddit thread full of low-effort arguments. A Wikipedia article with numerous references is more valuable than an LLM-written content farm page stuffed with keywords. The internet contains trillions of words, but the portion that is well-written, factually accurate, and diverse enough to train a capable model is a small fraction of it. Someone has to decide what gets filtered out. How much code versus how much natural language? How much scientific text versus how much casual conversation? These decisions determine the resulting model’s strengths and weaknesses, and they require human judgment that is difficult to fully automate.

The clearest demonstration of how much curation matters comes from Microsoft’s Phi series of models that are surprisingly capable for their size. The Phi team showed that a small model trained on carefully selected “textbook quality” data could outperform much larger models that were trained on bigger datasets with less careful curation. Reaffirming the theory that well-selected human training data is a multiplier on everything that occurs in the model training downstream.

The economics of pre-training data are shifting, and the technical challenge of assembling a pre-training dataset is becoming understood by frontier AI developers better and better. What’s emerging, however, is the difficult legal environment. AI developers have signed licensing deals worth hundreds of millions of dollars with organizations like Reddit and various news publishers. In the case of xAI, they went further in acquiring the entire social network X. The technical cost of curating pre-training data is going down, but the legal cost of the right to use high-quality text is going up. For pre-training, the emerging moat is legal access to premium content.

After pre-training, a model has absorbed enormous amounts of knowledge, but it’s still far from being a useful assistant.

Supervised fine-tuning (SFT) solves this by showing the model examples of the behavior you want: a user asks something, and a helpful assistant responds clearly. The model learns from these demonstrations in a similar way it learned during pre-training, but now it learns how to be an assistant, not just a human imitator.

What’s surprising is how little data this requires compared to pre-training. While pre-training consumes trillions of words, SFT might use just thousands of examples. But those examples need to be high quality. The quality of each individual demonstration matters far more than the total count, because each one is teaching the model what being helpful looks like.

Take DeepSeek R1, the Chinese model that shook markets in early 2025, as a good illustration of this process. DeepSeek first developed a model called R1-Zero using pre-training and reinforcement learning. R1-Zero exhibited impressive reasoning capabilities, matching OpenAI’s o1 on some benchmarks, but it had a problem: its outputs were messy, it occasionally mixed languages unpredictably, and sometimes produced unreadable text. The model could reason but struggled to communicate.

To fix this, DeepSeek fine-tuned their base model on just thousands of carefully selected examples. This cold-start data was small in volume but high in quality: the team collected readable reasoning traces, filtered out poorly formatted outputs, and refined results with human annotators. The effect was significant. The newly developed R1 kept the reasoning power of R1-Zero while producing outputs that humans could actually follow and use. Thousands of well-chosen examples turned a brilliant but incoherent system into a product that competed with the best models in the world.

The lesson here is that supervised fine-tuning is a stage where human taste and judgment have disproportionate leverage compared to the amount of data involved. The people writing and curating these demonstrations are making choices about what good output looks like, and those choices affect every response the model produces afterward.

The most recent development in how frontier models are trained is reinforcement learning with automated feedback. Instead of showing the model examples of good behavior, you put it in a situation where it can try things, observe results, and learn whether it succeeded or failed. If a model writes code, you can run the code and check if it passes tests. If it solves a math problem, you can verify the answer. The model gets a signal: that worked, or that didn’t. Over many iterations, it gets better.

This approach is producing rapid gains. METR, an AI safety organization, tracks how well models can accomplish software engineering tasks of increasing complexity (as measured by the amount of time it takes humans to complete them) and created a plot showing exponential progress of models. They found that the length of tasks that models can handle has been growing exponentially, roughly doubling every few months. A few years ago, frontier models could only handle several-minute-long tasks, but current models can work on genuine engineering problems that take hours to complete, and the majority of this improvement comes from RL.

RL training depends on expert human data. It needs an environment: a simulated world with complex realistic tasks and reliable feedback signals. In software engineering, you need a real codebase on a computer and a test suite that reliably tells you whether the code actually works. For RL training to be effective, you don’t need one such environment — you need thousands of them, each with different tasks and different levels of difficulty.

Building these environments is a new realm where human expertise becomes critical. The people designing RL environments for software engineering need to understand what real tasks look like, what makes a test suite reliable, and what constitutes a realistic set of tasks that teaches the model real software engineering skills. If your environments only contain simple bugs with obvious fixes, the model learns to solve simple problems. If they contain the kind of messy, ambiguous challenges that real software engineers face, the model develops skills that generalize. The quality of the environment directly determines the quality of the resulting model.

There are fears that synthetic data is making human input obsolete. AI models are increasingly generating training data for other AI models, and in some cases for themselves. Anthropic's Constitutional AI, which is their version of preference training, is a good example: instead of hiring thousands of human raters to judge which model response they like more, Anthropic wrote a set of principles (the "constitution") and let the AI rate its own outputs by testing them against the constitution. This dramatically reduced the need for human raters compared to other preference training approaches, like RLHF. However, human input didn't disappear; it just became more high-level. Instead of rating individual responses, as other companies do, humans designed the principles by which responses should be rated by AIs. The routine labor became automated, but humans still designed the whole process.

Synthetic data has a deeper limitation, though. Model-generated text is less diverse than human-generated text, and this compounds as models are trained on the outputs of models that have been trained in the same way. Researchers have studied what happens when models are trained this way across multiple generations, and found that each generation loses more of the unusual and rare content that exists in human-written text. After several generations, the output converges toward a bland, repetitive mean, and performance degrades—a phenomenon researchers call model collapse. One study demonstrated this with image generation: models trained on AI-generated faces gradually lost the ability to produce faces of underrepresented ethnicities, converging on a narrow set of features from the majority of training examples. Synthetic data has its uses, but it cannot replace the diversity that human-generated data provides.

The parts of AI training that are easiest to automate are getting automated. Automated reward signals can tell a model whether its code compiles without any human in the loop.What remains is the work that requires judgment:deciding which data belongs in a pre-training corpus,writing the demonstrations that teach a model what a good assistant would respond, designing RL environments that reflect the complexity of real-world tasks rather than presenting a model with toy problems.

This has a concrete implication for anyone evaluating companies that provide training data to AI developers. “Training data” is not one market. There are several markets with fundamentally different dynamics, and the direction each market is heading matters more than its current size.

Pre-training data is becoming well-understood technically. The hard problems are legal, not technical: obtaining the right to use high-quality text as copyright litigation intensifies as licensing deals get more expensive. Companies in this space are building moats out of content access and legal agreements, not out of technical expertise that competitors can’t replicate.

Other stages of training are moving in the opposite direction. The technical challenges are growing. Building thousands of high-quality RL environments for professional services proves to be a genuinely hard problem that requires deep knowledge across domains, and researcher’s appetite for this kind of data increases as they push harder on post-training. The moat here is expertise: understanding what makes a training environment reliable enough to produce models that perform well on real tasks.

The broad trend is that as we need to train models to become more and more capable, human contribution to AI training is concentrated into higher-skill, higher-leverage roles. The value created by those who remain is increasing, because they are doing the work that the models themselves cannot yet do.

Training data for AI is not becoming less human. It is becoming human in a different way. The volume of human labor in the pipeline may decrease as synthetic data and automated feedback take over routine tasks. But the value of the human work that remains is growing, because it sits at the points in the pipeline where judgment, expertise, and taste determine whether a model is merely functional or genuinely capable. For anyone trying to understand where the AI industry is heading, this shift matters: the companies that will be hardest to replicate are not the ones sitting on the largest piles of data, but the ones whose people know how to create the data that models cannot generate for themselves.