Ask ten people whether a customer message is “urgent,” and the answers may be surprisingly varied. “My account is locked and I need to pay someone today” feels urgent. “I was charged twice last week” might be urgent to the customer and routine to a support queue. “Your app is broken” is certainly urgent to somebody, but the message supplies almost no useful detail. The raw text does not contain a tiny, pre-installed urgency label waiting to be discovered. It contains a situation that someone must interpret.
Data annotation is the process of adding labels or judgments to raw examples so an AI system can learn what to recognize, generate, rank, or avoid. A label can be a category such as “billing issue,” a text span identifying a person’s name, a box around an object in an image, a severity score, an ideal answer to a prompt, or a choice between two model responses. Annotation is how a pile of observations becomes a learning signal for supervised learning.
That makes annotation more consequential than it first appears. The interface may offer a few buttons and a progress bar, but the real work is deciding what each button means. A model trained on the resulting labels does not learn the world in the abstract. It learns the version of the world that the label definitions, examples, exceptions, and reviewers made legible.
A Label Is a Decision Rule
People often use “label” as if it means a fact taped onto an example. That description works for some tasks. A photograph may show a particular traffic sign. An audio clip may contain a particular word. Yet even apparently simple tasks come with choices about boundaries, context, and purpose.
Consider a support message that says, “I cannot access my account, and I also need an invoice.” Is that one label, two labels, or a primary issue plus a secondary tag? The answer depends on what the downstream system must do. A routing model may need multiple intents. A reporting dashboard may need one main reason for contact. A language model answering the customer needs a different kind of annotation entirely: it may need the relevant account state, a safe response pattern, and an escalation rule.
An annotation schema defines the allowed shape of those answers. It might specify categories, nested tags, spans of text, ordered ratings, response pairs, or structured fields. Annotation guidelines explain how to apply that schema: what counts, what does not, how to treat edge cases, which evidence takes priority, and what to do when the example cannot be determined confidently. The schema is the map legend; the guidelines are the part that explains what to do when the terrain has been paved over, renamed, or flooded.
A useful guideline does more than list class names. It includes positive examples, exclusions, boundary examples, tie-break rules, and an honest “unclear” path. Without that last option, an annotation task quietly asks people to invent certainty whenever the example refuses to cooperate. The result may look wonderfully clean in a spreadsheet and behave like a rumor in a training run.
This is why annotation begins with the model’s intended decision, not with the number of rows in a queue. If a system is supposed to detect refund requests, the team needs to define whether a message asking about a delayed charge belongs in refunds, payments, or a category for account investigation. No amount of post-hoc aggregation can turn an unstated policy into a consistent label.
A broad review of text-dataset practice separates the scheme, which defines the labels, from the guidelines, which define how to decide among them. It also found that quality management varied widely across 591 dataset papers, with guideline improvement, annotator training, and iterative pilots reported less often than their importance would suggest (Klie et al., 2024). The less glamorous truth is that a good instruction sheet can improve a model more than a surprising amount of model tinkering.
The Rulebook Meets Reality
The first draft of annotation guidance is rarely the final one. That is not a failure of planning. It is what happens when a neat concept encounters actual language, images, audio, and human behavior. Edge cases are not interruptions to the project. They are the part of the project that tells you what the task actually is.
A sensible annotation workflow starts with a small, deliberately varied pilot. Include routine examples, obvious negatives, borderline cases, incomplete inputs, and examples that combine multiple categories. Then compare decisions and ask a specific question: did annotators disagree because one person made a mistake, because the guideline was unclear, because the schema lacks a useful category, or because the case supports more than one reasonable interpretation?
That review creates calibration, the process of aligning people’s interpretations before or during production annotation. Calibration is not a pep talk in which everyone promises to be more consistent. It means comparing concrete examples, surfacing competing rationales, recording the decision, and updating the guidelines so the next reviewer does not have to rediscover the same ambiguity alone.
Pradhan and colleagues tested a FIND-RESOLVE-LABEL workflow in which people first identified examples that made task instructions ambiguous, a requester resolved selected examples, and the clarifying cases were added back into the guidelines. Their experiments found improved annotation accuracy when later workers used the revised instructions (Pradhan et al., 2022). It is a useful reversal of the usual instinct. Do not wait for the exceptions to ambush the production queue. Go looking for them.
The guidelines should be versioned, too. If “escalate immediately” changes from a two-hour response window to a same-day response window, the label definition has changed. Mixing examples from both policies without recording the transition makes the dataset look stable while teaching the model two quietly incompatible rules. That is a very efficient way to manufacture a disagreement that nobody can explain later.
Agreement Is a Clue
When two independent annotators label the same item, inter-annotator agreement measures how often their decisions are compatible. It is a useful diagnostic. It tells a team whether people applying the current instructions tend to converge. It does not, by itself, prove that the instructions describe the right task, that the data represents the intended environment, or that all disagreement is a mistake.
This distinction matters because annotation tasks come in different species. A transcription task can often aim for one correct string. A task involving sentiment, offensiveness, helpfulness, appropriateness, or risk can include legitimate variation. An annotator with direct domain expertise may interpret a phrase differently from someone with a generic instruction sheet, and that difference may contain information the project ought to preserve.
Artstein and Poesio’s survey of agreement in computational linguistics remains a useful reminder that agreement statistics need to match the annotation setup and be interpreted in context (Artstein & Poesio, 2008). High agreement can mean the guidelines are clear. It can also mean the labels are too coarse to capture what matters. Low agreement can reveal inadequate training or fuzzy definitions. It can also reveal that people genuinely see the example differently.
That is why “get more labels and take the majority vote” is not a complete quality strategy. Majority voting is an aggregation method, not a philosophy of truth. Davani, Díaz, and Prabhakaran found that collapsing subjective annotations into a single majority label can erase systematic differences in judgment; their multi-annotator approach preserved those differences and matched or outperformed aggregated-label baselines across several tasks (Davani et al., 2022).
For some tasks, retaining soft labels, meaning a distribution of judgments rather than one winner, is the more honest representation. Uma and colleagues’ survey found that when datasets contain many judgments from high-quality annotators, training directly on soft labels can outperform training on aggregated labels (Uma et al., 2021). This does not mean every project needs five annotators per example and a dissertation on human subjectivity. It means the annotation design should fit the kind of uncertainty the model is expected to manage.
The table points to a larger lesson: disagreement should trigger investigation, not reflexive cleanup. A split on the same tricky example might expose an undefined boundary. A steady difference between two experts may expose separate valid perspectives. A consistently odd rater could be confused, but they might also be noticing a condition that the original schema forgot to name.
Quality Control Is a Feedback Loop
The best annotation programs do not rely on one final inspection, any more than good software teams rely on one enormous test run the night before launch. They build several small checks into the process: pilot batches, calibration examples, periodic double-annotation, reviewer samples, automated consistency checks, and a record of why contested cases were resolved in a particular way.
A gold set is a carefully reviewed collection of examples used to calibrate or assess annotation work. Gold sets are useful when the target is well-defined and the review process is documented. They become misleading when they pretend to provide a timeless answer key for a subjective judgment. If a quality reviewer marks a response “unhelpful,” the important question is not only whether the reviewer agrees with the gold label. It is whether the rubric explains what helpful meant in that decision.
Adjudication is the process of resolving, documenting, or deliberately preserving different annotations after review. For a factual task, adjudication may mean checking evidence and correcting an error. For a guideline gap, it may mean revising the rule and rechecking earlier examples. For a subjective task, it may mean retaining multiple views, storing a rationale, or narrowing the dataset’s claim. These are different operations, and treating them all as “cleaning” makes a dataset sound tidier than it is.
Automated analysis can help focus attention. Confident learning uses model predictions and observed labels to estimate label noise and rank examples that may warrant inspection (Northcutt et al., 2021a). That makes it a valuable triage tool. It does not turn an algorithm into the final authority on which human judgment was correct. A model flagging a label as surprising may have found a genuine error, a rare but valid case, or a weakness in the model’s own assumptions.
The stakes extend to evaluation, not just training. A study of ten widely used computer-vision, language, and audio test sets estimated at least 3.3% label errors on average, showing how noisy annotations can distort the benchmarks used to compare models (Northcutt et al., 2021b). This is one reason model evaluation is never only about choosing a metric. A model can be graded against a flawed answer key with impressive mathematical precision.
When Better Means Preferred
Annotation becomes especially interesting for large language models (LLMs) because many of the desired outputs are not simple categories. A reviewer may write an ideal response, grade an answer against a rubric, flag an unsupported claim, score style or safety, or rank two candidate completions. These labels help define behavior after broad pretraining has already taught the model a great deal about language.
Preference data contains an input, competing candidate outputs, and a judgment about which output is better under a stated criterion. It is central to RLHF, but the important annotation question arrives earlier: better according to what? A shorter answer may be preferable for a busy user and unhelpful for someone asking a complex technical question. A safe refusal may be correct and still be needlessly opaque. Factual accuracy, task completion, tone, policy compliance, citation quality, and concision can all point in different directions.
Ouyang and colleagues used labeler demonstrations and rankings of model outputs to fine-tune language models toward instruction following (Ouyang et al., 2022). The lesson for annotation design is not merely “collect rankings.” It is to make the rubric legible. Annotators need to know how competing goals trade off and how to handle two imperfect answers. Otherwise, “human preference” becomes a polite name for whatever someone selected at 4:57 p.m. after reviewing a long queue.
The same issue appears in training datasets for instruction following. A single ideal response can hide multiple reasonable ways to answer a request. A careful dataset may therefore include rubrics, alternate acceptable responses, structured attributes, or preference pairs rather than insisting that every prompt has one sacred completion. The training signal gets richer when it records the reasons a response is useful, not just the final thumbs-up.
Automation Can Carry a Clipboard
Modern annotation workflows increasingly use models to propose a category, extract a span, draft a rationale, identify a potential conflict, or prioritize examples for review. Pre-annotation means a model supplies that provisional first pass for a human to inspect. It can turn repetitive labeling into verification work and reserve expert attention for difficult cases, but it also changes the psychology of review.
Wang and colleagues’ model-in-the-loop research used LLMs as pre-annotation assistants, real-time aids, and judges alongside professional annotators. In their studied multimodal workflows, the approach reduced handling time and improved both quality and annotator experience (Wang et al., 2024). The useful pattern is not “let the model label everything.” It is “give people a better starting point, then measure whether the starting point changes the errors they make.”
A plausible suggestion can anchor a reviewer even when it is wrong. A generic model definition can conflict with a project-specific rule. And a system trained on labels generated by a similar system can reinforce the same blind spot with the serene confidence of a meeting that has confused agreement with progress. Strong workflows protect against this by keeping independent human labels for a sample, auditing where human and model judgments diverge, and reviewing error clusters rather than only overall acceptance rates.
Weak supervision offers another way to create labels at scale. Instead of hand-labeling each example, domain experts write rules, heuristics, or external lookups that label examples noisily. Snorkel demonstrated how such labeling functions, even when their accuracy and correlations are unknown, can be combined to create training data (Ratner et al., 2017). The human judgment has not disappeared. It has moved upstream into the rule design, the assumptions about errors, and the review of what the rules systematically miss.
Automated preference generation has a similar tradeoff. Shi, Chen, and Zhao describe a pipeline for producing and filtering safety preference data with models (Shi et al., 2024). Such methods can expand coverage, especially where manual work is scarce. They also make provenance and targeted human audits more important, because the source model, filtering criteria, and hidden rubric have become part of the annotation policy.
The Specification Hiding in Plain Sight
Teams sometimes describe annotation as if it sits below the interesting work: someone else collected the data, someone else will train the model, and the labelers are merely filling in the middle. In practice, annotation is where a product’s expectations become instructions a model can optimize.
A well-designed annotation process makes its claims inspectable. It says what a label means, who applied it, which examples made the rules wobble, what disagreement revealed, how decisions were reviewed, and what uncertainty remained. A weaker process can still produce millions of labels. It just leaves future users to infer the specification from model behavior, which is a slow and expensive way to read a rulebook.
Good annotation does not promise to eliminate judgment. It gives judgment a structure, a vocabulary, and a trail of evidence. That is how a model receives more than answers. It receives a usable account of what those answers were supposed to mean.
