A five-star scale has the look of precision. It turns a reaction into a number, puts a decimal point behind it, and gives everyone the pleasant impression that a difficult judgment has been domesticated. Then one reviewer gives a response four stars because it is “quite good,” another gives it four because it “barely avoids three,” and the spreadsheet carries on as though nothing happened. The spreadsheet is not wrong. It is just magnificently uncurious about what the numbers mean.
Human preference data is a collection of structured judgments showing which of two or more AI outputs people prefer, or how outputs score against defined criteria. A simple record contains a prompt, two candidate responses, and a choice of chosen over rejected. A richer one might include separate scores for factuality and helpfulness, a written critique, the reviewer’s role, the rubric version, or an “I cannot decide” option. It gives large language models (LLMs) a signal for behavior that is not neatly captured by one objectively correct answer.
The important word here is not human. It is preference. A preference record is a tiny measurement device for an otherwise slippery idea: which response is more useful, safer, clearer, more accurate, more appropriate, or more likely to accomplish the job. The device can be well calibrated, poorly calibrated, pointed at the wrong thing, or quietly measuring a reviewer’s taste in prose. That is why the dataset deserves more attention than the deceptively tidy chosen and rejected columns suggest.
A Vote Is Not Yet a Measurement
Pairwise comparisons are the usual starting point because people are often better at answering “Which of these two is better?” than “Give this answer an absolute score from one to ten.” The question is local. It asks a reviewer to compare two concrete candidates under a defined task, rather than to summon an internal universal scale that may be operating on coffee, mood, and the previous twelve answers they saw.
A pairwise comparison presents the same prompt with two outputs and asks which one wins. In the standard form, it produces a preference record: (prompt, chosen response, rejected response). That compact structure works well for ranking systems and for alignment methods because it says not merely that one answer was acceptable, but that it was better than a particular alternative.
The approach has a longer history than chatbots. Human comparisons between short segments of behavior allowed reinforcement-learning agents to learn goals that were difficult to express as a hand-written reward function, while requiring feedback on only a small portion of the agent’s interactions (Christiano et al., 2017). The core insight carries over to language: it is easier to compare two summaries than to formalize “summarizes well” as a single equation and hope the equation knows what it is doing.
Still, a comparison is useful only when it asks a meaningful question. If one response is competent and the other reads like it was generated during a turbulent landing, the label is cheap but not very instructive. If both responses are flawed in different ways, the reviewer needs to know which flaw matters more. And if the rubric is absent, every rater brings a private version of “better” to the task. The resulting data may be consistent enough to train a model, which is not the same thing as being clear enough to defend.
Researchers studying preference datasets describe a common minimum structure as a prompt, two responses, and an annotation identifying the preferred response (Shen et al., 2024). That is the skeleton. A production-quality dataset needs enough context to explain the skeleton’s decisions later.
The Rubric Carries the Real Policy
A rubric tells a reviewer how to decide. It is the part of the dataset that answers the awkward but essential question: better for what?
An assistant can be concise, accurate, friendly, careful about uncertainty, safe, and directly responsive. It can also be all of those things at once only on unusually cooperative prompts. Most real comparisons involve tradeoffs. A short answer can be pleasantly direct and omit the one caveat that changes the decision. A detailed answer can be factually sound and bury the useful sentence somewhere in paragraph six. A safety-conscious answer can refuse correctly or refuse from a distance of three postal codes.
A rubric turns those tradeoffs into an inspectable policy. It may say that correctness beats brevity, that a refusal must explain a safe next step, that citations matter for a research task, or that a coding answer is incomplete until it respects existing interfaces. Without those rules, preference collection becomes a contest in which reviewers are asked to grade the assignment before they have been told what subject they are teaching.
The most durable rubrics define a limited set of criteria, explain priorities, include boundary examples, and distinguish a hard constraint from an ordinary preference. “Do not invent a medical claim” is not merely a style preference that loses by two points to a warmer tone. “Use fewer words” is rarely a reason to omit the answer. Such distinctions make the data more useful because they reveal why one response won.
Multi-attribute feedback helps when the goal is genuinely multi-dimensional. UltraFeedback, for example, assessed instruction following, truthfulness, honesty, and helpfulness separately, supplied detailed scoring criteria, and asked for critique alongside scores (Cui et al., 2024). Those exact categories are not a universal recipe. The useful lesson is that a thumbs-up cannot gracefully carry five different judgments on its back forever.
Systems that validate agent-built work, such as Sandgarden's Gloria.dev, make the same practical point in another setting: an output cannot be checked against an unstated specification. Preference data needs that specification too. Otherwise, the dataset is a record of vibes with excellent row counts.
What a Preference Record Should Preserve
A winner is the beginning of a record, not the whole record. The fields around it determine whether a team can later understand a pattern, correct a bad assumption, or tell a product-specific judgment from a general one.
Notice that this is not paperwork for paperwork’s sake. Each field protects against a different kind of fiction. Source versions expose whether a model’s polished style is beating a weaker model on presentation alone. A rubric version prevents a policy update from masquerading as inconsistency. An abstention field stops a reviewer from declaring a winner when the pair contains two different kinds of bad answer.
The record also needs enough provenance to support an argument with a skeptical future teammate, auditor, or customer. “The reward model likes it” is not an explanation. It is an invitation to ask what the reward model saw, who supplied the comparisons, and whether that group was asked a coherent question.
The Candidates Choose the Lesson
The outputs shown to reviewers are not neutral raw material. They are an experimental design choice. Candidate generation decides which distinctions the dataset will teach a model to notice.
Easy comparisons have a role. They can catch broken prompts, establish that reviewers understand the task, and prevent a quality program from becoming unnecessarily theatrical. But a dataset made mostly of obvious wins produces a model that learns obvious wins. It may become skilled at preferring a fluent answer over nonsense and still fail when deciding between a useful answer with one missing caveat and a longer answer that answers a slightly different question.
The more revealing pairs make a single tradeoff visible. One candidate might be accurate but miss a required output format. Another might be concise but overstate uncertainty. A third might be helpful on the surface and rely on a claim that cannot be supported. These are not “gotcha” examples. They are the real decisions a system will face once it leaves the tidy world of demonstrations.
Candidate diversity matters, too. If every rejected response comes from the same weak model, the reward model can learn surface clues about that model’s style rather than the intended objective. UltraFeedback sampled responses from a pool of different model families and capability levels partly to reduce that kind of spurious association (Cui et al., 2024). It is the same reason a science experiment should not put every treatment group in the same suspiciously sunny corner of the greenhouse.
Dataset size does not rescue an unhelpful candidate design. In one study of preference datasets, larger collections did not reliably improve out-of-domain reward-model performance, while dataset composition and relevance could matter more than simply adding examples (Shen et al., 2024). The practical translation is blunt: ten thousand well-chosen disputes may teach more than a hundred thousand occasions for everyone to agree that the answer with facts is better than the answer with gibberish.
Whose Preference Is It?
The phrase “human preference” can sound like a single smooth curve. It rarely is. A trained domain reviewer, a frequent user, a safety specialist, a community moderator, and a randomly selected crowd worker may give different answers for reasonable reasons. The dataset’s rater population is therefore part of its meaning, not a background implementation detail.
A team might want a shared organizational policy, a domain-expert standard, the distribution of a target user population, or a personalized view of one user’s recurring choices. These are different targets. They require different sampling, different data fields, and different claims about what the model has learned.
The difference becomes obvious in high-stakes or subjective work. A medical assistant should not learn from a popularity contest when a clinical standard is required. A writing assistant may need to adapt to one user’s taste in directness without treating that taste as a universal preference. A safety rule may deliberately override ordinary user preference. Calling all of these “human feedback” is convenient in the same way calling a kitchen, a chemistry lab, and a tire shop “places with tables” is convenient.
Disagreement is not automatically noise. Research on human-labeled preference datasets identifies task underspecification and response style among frequent sources of rater divergence, challenging the assumption that every disagreement is a corrupted version of one true ranking (Zhang et al., 2025). A split vote may expose an unclear rubric, a missing user context, a real value conflict, or an output whose merits depend on a stated audience.
A useful response is to inspect the split, not merely erase it. Teams can revise the rubric, add a rater population field, collect rationale, preserve multiple judgments, or separate a hard safety constraint from preferences about tone. The correct move depends on the objective. Forcing every disagreement into a majority vote may make the CSV cleaner, but it can also sand off the very distinction the product needs to respect.
Personalization makes the issue more visible. Work on preference descriptions inferred from user writing has found that generic descriptions can miss individual variation, while iterative inference and verification across several samples can capture more nuanced preferences (Aroca-Ouellette et al., 2025). This is a reminder that a broad preference dataset can be useful without being a substitute for every person’s preference.
A Dataset Can Say More Than Chosen and Rejected
Binary comparisons are popular because they are manageable. They are not the only format worth collecting.
A ranking asks a reviewer to order several candidates. It can reveal more information per prompt, but it also increases cognitive load and makes inconsistent orderings easier to produce. A scalar rating asks for a score, often on separate dimensions such as factuality or completeness. It is useful when the dimensions themselves matter, though different reviewers may use the scale differently. A critique records a written explanation of what failed and how to improve it. It is harder to standardize, but exceptionally useful when a team needs to inspect failure patterns rather than merely count losses.
Some products can use actual edits as a preference signal. If people repeatedly rewrite an assistant’s explanation to add a caveat, remove boilerplate, or change a tone, those revisions say something valuable. They also say something about the editor’s habits, deadlines, and tolerance for cleanup. Observed behavior should be treated as a signal with context, not as a magical direct line to intent.
There is no universally richest format. A survey of preference learning distinguishes relative relationships, which are natural for people to express but comparatively sparse, from absolute properties, which can provide more detail but are harder to collect consistently (Jiang et al., 2024). Hybrid records can be especially useful: a pairwise choice for the downstream optimizer, attribute ratings for diagnosis, and a short rationale for audit.
The choice should follow the question. If the task is “Which of these two responses better follows a strict schema?”, a pair is often enough. If the task is “Which response better supports a difficult customer while remaining accurate and policy-compliant?”, a single winner with no stated reason leaves too much of the important work hidden.
The Data Keeps Working After Collection
Once preference records exist, several systems can use them. A reward model learns to predict which response is likely to win. RLHF can use that learned signal to guide further training. DPO instead uses chosen and rejected responses directly in its optimization objective, avoiding a separate reward-model training stage (Rafailov et al., 2023).
Those mechanisms matter, but they do not change the upstream fact: each is only as useful as the distinctions preserved in the data. InstructGPT combined labeler demonstrations with rankings of model outputs, showing how the same broad goal can need both examples of desired behavior and comparative judgments between plausible alternatives (Ouyang et al., 2022). The dataset does not have to serve one method forever. It may support selection among candidates, model evaluation, targeted regression tests, or later fine-tuning as product needs change.
That is one reason to retain context and rationale rather than publishing a bare pairwise file. A team may initially train a reward model, then discover that the more urgent use is diagnosing why the assistant is overly verbose, too quick to refuse, or unhelpfully certain. Data that records only the winner can answer fewer questions when the system changes.
Audit the Objective, Not Just the Labels
Preference-data quality is often reduced to agreement rate or the number of pairs collected. Both are useful, neither is enough. A better audit asks whether the collection matches the decision the system will make.
Start with objective fit. Can a reviewer determine what the product needs from the rubric alone? Then examine coverage. Does the dataset include routine cases, difficult boundaries, high-risk contexts, and the relevant kinds of user request? Check candidate validity. Are the responses comparable, and does the collection design accidentally make one model family’s style a proxy for quality?
Next comes provenance. A team should be able to trace prompt source, candidate generator and version, rubric version, collection date, rater procedure, and any model assistance used during review. Preference data changes meaning when the policy changes, when the response generator improves, or when the intended user population shifts. A dataset with no such history is not timeless. It is merely undocumented.
Finally, test whether the data improves behavior where it matters. RewardBench evaluates reward models using prompt-chosen-rejected trios across chat, reasoning, and safety, including subtle comparisons with verifiable reasons for one answer to win (Lambert et al., 2024). That kind of evaluation is useful because matching easy in-distribution choices is a low bar. A reward model should also distinguish a polished mistake from a plain but correct answer when the actual task demands it.
Careful data can be efficient as well as principled. HelpSteer2 reported strong reward-model results with ten thousand multi-attribute response pairs, a useful counterweight to the reflex that more pairs must always be better (Wang et al., 2024). The lesson is not that ten thousand is a magic number. It is that preference data should earn its volume by capturing relevant judgments, not by reproducing the easiest one.
Human preference data gives an AI system a way to learn what people want when a conventional answer key runs out of road. It works best when the team treats it as a designed measurement system: specify the objective, choose candidates that expose meaningful tradeoffs, know whose judgments are represented, retain uncertainty, and test the results against the behavior that matters. The model will take the dataset seriously. It has no choice. The people building the dataset should do the same.
