An agreement score can be both reassuringly high and quietly misleading. When nearly every item belongs to the same category, two reviewers can match on almost everything without demonstrating much shared judgment. A low score can be misleading in the opposite direction: people may be responding to a genuinely ambiguous, subjective, or under-specified task, not making careless mistakes. Inter-annotator agreement is a family of measures that quantifies how consistently independent people apply the same annotation scheme to the same items. The number is useful precisely because it can expose these situations. It just cannot tell you which one you have without a little detective work.
That makes agreement less like a report card and more like a dashboard warning light. A light tells you to look under the hood. It does not identify the loose hose, write the repair manual, and certify the car for a cross-country trip. For machine learning teams, the same restraint matters. Agreement can reveal whether labels used to train or evaluate a system are reproducible. It cannot, by itself, prove that the labels describe the right concept, that the raters represent the intended audience, or that a majority vote has discovered a single truth.
An Instrument Rather Than a Verdict
The most direct number is observed agreement, often called percent agreement: how often reviewers give the same label to the same item. It has one great virtue, which is that anyone can understand it. If two reviewers matched on 82 of 100 examples, their observed agreement is 82%. Report it. Hiding it behind a more elaborate coefficient does not make it disappear.
Its limitation is equally straightforward. Some matching would occur even if reviewers did not share a useful understanding of the task. In a dataset where 98% of messages are not urgent, two people who nearly always choose “not urgent” will agree often. That does not mean they have learned the difficult boundary between a routine delay and an escalation. It may simply mean that the data offered them an easy way to be right together.
Chance-corrected agreement statistics try to separate observed matching from the matching their assumptions would expect by chance. Cohen’s kappa is the famous version for two reviewers assigning nominal categories. It compares the observed agreement to expected agreement derived from each reviewer’s overall label distribution. If reviewers use the common category frequently, kappa gives less credit for matching there (McHugh, 2012).
This is a helpful correction, not a magic trick. Every agreement coefficient carries assumptions about the task, raters, labels, and missing responses. The number becomes meaningful only after the team has decided what kind of consistency it wants to measure. The central distinction is reliability versus validity. Reliability asks whether the current process produces similar results when repeated. Validity asks whether that process captures the thing the team actually intends to measure. A beautifully consistent scheme can still classify the wrong thing with industrial efficiency.
That distinction has been central to computational linguistics for years. Agreement statistics need to fit the annotation setup and be interpreted within the purpose of the study, rather than treated as portable labels for “good data” or “bad data” (Artstein & Poesio, 2008). The temptation to skip this work is understandable. A single coefficient fits neatly in a paper table. A discussion of the measurement design requires admitting that the table is not the interesting part.
What the Coefficients Are Asking
A sensible choice starts with the annotation design, not with the statistic that happened to be available in the first software menu. With two reviewers and mutually exclusive categories such as billing, technical, or other, Cohen’s kappa is often a reasonable place to start. When several reviewers assign the same nominal categories to each item, Fleiss’ kappa extends the chance-corrected approach across the group.
The situation changes when labels have an order. A disagreement between “mostly clear” and “clear” is different from a disagreement between “mostly clear” and “unintelligible,” even if both are technically mismatches. Weighted kappa can give partial credit for nearby ordinal ratings, provided the team states and justifies the weighting scheme. Otherwise, the distance between categories is being decided by a formula nobody has introduced to the readers, which is an odd way to set policy.
For projects with multiple reviewers, varying numbers of ratings per item, missing judgments, or data beyond nominal categories, Krippendorff’s alpha is often more flexible. It works by comparing observed disagreement with disagreement expected under its chance model, and it can use an appropriate distance function for ordinal, interval, or ratio data. That flexibility is why alpha appears so frequently in language and content-analysis work. It is also why a report should state the data type and distance function, rather than assuming that the Greek letter will handle the explanation by itself.
Some jobs do not involve one label per row at all. A named-entity task asks people to mark text spans. A segmentation task asks them to place boundaries. A quality-review task may ask for a continuous score. Pairwise span overlap, boundary-oriented measures, and forms of the intraclass correlation coefficient can be more appropriate in those cases than a single-label kappa. A screw driver is not a bad tool because it cannot make soup; it is just a poor reason to keep using a screw driver in the kitchen.
Recent NLP guidance makes the selection logic explicit: account for the data type, number of reviewers, label imbalance, missingness, and whether the task needs absolute agreement, ranking consistency, or overlap between structured annotations (James, 2026). In a simulation study of nominal labels, Fleiss’ K and Krippendorff’s alpha behaved similarly with complete data, while alpha handled missing ratings more gracefully; the authors also found that bootstrap confidence intervals were preferable to the usual asymptotic interval for Fleiss’ K in their settings (Zapf et al., 2016).
The table is a map, not a law. Two annotation projects can both be called “classification” while needing different measures because one uses a stable expert panel and the other gathers a sparse set of crowd ratings. The useful question is not “Which metric is standard?” It is “What process would make this score interpretable to someone who did not build the dataset?”
When Kappa Seems to Disagree With You
The prevalence paradox is where agreement statistics become unexpectedly entertaining. Imagine a rare safety category. Reviewers agree that 990 of 1,000 ordinary requests are ordinary, then disagree on several of the ten unusual ones. Observed agreement can look excellent because the dominant class supplies hundreds of easy matches. Kappa can look surprisingly low because, given each reviewer’s tendency to choose the dominant category, much of that matching is expected under its chance model.
Neither number is necessarily broken. They are answering different questions. Observed agreement describes the visible matching in the data. Kappa asks how much matching remains after a particular estimate of chance agreement. When categories are heavily imbalanced, the estimate can dominate the result. The right reaction is not to declare one number the winner in a tiny statistical cage match. Inspect the category frequencies, the underlying agreement table, the number of positive examples, and the kind of error that matters for the task.
A study of three evaluators illustrates the issue: high observed agreement on strongly prevalent categories came with very low kappa values, while Gwet’s AC1 produced values closer to the observed concordance (Zec et al., 2017). That does not establish AC1 as the universally correct replacement. It shows why teams should report their raw agreement, label prevalence, and a justified chance-corrected measure together. A coefficient should earn its authority by making its assumptions visible.
Rater bias can complicate the picture further. One reviewer may use the uncertain label more freely, another may reserve it for only the hardest cases, and a third may almost never choose it because the guideline makes the category feel like an admission of defeat. The aggregate score can conceal that pattern. A confusion matrix, item-level review, and reviewer-specific label counts often reveal more than a decimal with three places after it.
Ordinal work adds another twist. If reviewers rate toxicity on a five-point scale, treating a one-step difference the same as a four-step difference discards information that the task deliberately collected. Weighted forms of alpha or kappa can represent that distance, and experimental work on emotion, opinion, and coreference annotation has found weighted Krippendorff’s alpha especially useful for multi-reviewer ordinal settings (Antoine et al., 2014). The weights should reflect a reasoned view of the scale, not a late-night preference for quadratic functions.
A Low Score Has More Than One Cause
The same agreement score can come from several very different projects. This is where teams get into trouble by treating the number as a diagnosis instead of a symptom.
Sometimes the trouble is execution. A new reviewer may misunderstand the interface, lose focus during repetitive work, or lack the domain context needed for the category boundary. That is a reviewer-level pattern, and it calls for training, a calibration set, targeted feedback, or a reconsideration of who should label the task. A broadly low score across experienced reviewers suggests a different problem.
More often, the annotation scheme has left a gap. The guidelines may define a category clearly at the center and become foggy at the edge. Disagreement then clusters around a specific kind of item, such as implicit requests, mixed sentiment, indirect threats, or claims with partial evidence. The team should look at those examples together, record the competing interpretations, decide what the downstream system needs, and revise the guidance. A guideline is a living instrument, not a legal tablet lowered from a mountain.
A third possibility is that the item itself genuinely lacks enough information. Context can be absent, a sentence can carry two reasonable readings, or a rating may depend on the audience. Adding more labels will establish which interpretation is most common, but it will not necessarily create the missing context. Here, retaining the disagreement, adding an uncertain path, or gathering more information can be more honest than forcing consensus.
There is evidence that disagreement can help locate these repair opportunities. The FIND-RESOLVE-LABEL workflow asks reviewers to identify examples that reveal ambiguity in task instructions, resolves selected cases, and feeds the clarifying examples back into the guidance; the authors reported improved annotation accuracy after the revision (Pradhan et al., 2022). That is a useful operating model for agreement studies: use disagreement to improve the process before using it to judge the people in the process.
This is also why an agreement calculation should occur during a pilot, not solely after a large annotation budget has already become a historical artifact. Reviews of 591 text-dataset papers found recurring weaknesses in agreement practice, including point estimates with little interpretation, too-small samples, and underreported quality procedures (Klie et al., 2024). A number calculated at the end of an unrecoverable workflow is still technically a number. It is just less helpful than one that arrives while the team can act on it.
Consensus Is a Design Choice
Some tasks truly aim at one determinate answer. If the work is transcribing a standardized identifier, checking whether a documented policy condition is present, or applying a well-specified technical rule, disagreement is usually a signal to refine the instructions, retrain reviewers, or investigate errors. A systematic resolution process is valuable in these cases, especially when the final dataset is meant to establish a single ground-truth label (Oortwijn et al., 2021).
Other tasks are about judgments that can vary with experience, community, context, or intended user. Labels such as offensive, persuasive, helpful, empathic, fair, or safe may contain a shared core alongside legitimate variation. Treating every split as a flaw can erase the information that the model needs to handle carefully.
Research on offensive-language data makes the point sharply. Hard, low-agreement instances were not necessarily poorly annotated, and selecting material across agreement levels affected classifier performance and robustness (Leonardelli et al., 2021). Work on evaluation similarly argues that disagreement can arise from the annotator, the item, and its context, even when a project is tempted to demand one clean gold answer (Basile et al., 2021).
The practical choice is not between “take every vote as sacred” and “majority vote forever.” A team can preserve a soft label, meaning the distribution of reviewer judgments, alongside an adjudicated label for a specific operational use. It can model uncertainty, record reviewer groups, or use disagreement to route difficult cases to human review. In several subjective classification tasks, models that retained reviewer-specific judgments matched or exceeded approaches trained on an aggregated majority label (Davani et al., 2022). The correct choice follows the intended decision, not a ritual attachment to tidy rows.
Results That Someone Else Can Trust
An agreement score becomes much more useful when another person can reconstruct its setting. State the unit of annotation, task instructions, label types, number of items, ratings per item, reviewer population, and whether the same reviewers saw every example. Report the category distribution and observed agreement beside the chosen coefficient. Explain the coefficient’s fit to the design, including weights or distance functions for ordinal and structured work.
Include uncertainty. A point estimate based on a small pilot can imply an absurd level of precision. Confidence intervals remind readers that the coefficient is an estimate, and they make it easier to distinguish a stable finding from a narrow sample that happened to look tidy. Where missingness, prevalence, or reviewer overlap matters, say so. The number should come with its wiring diagram, not only a light on the dashboard.
Breakdowns matter, too. Overall agreement can hide a category with almost no consistency, a reviewer whose labels drift over time, or a family of boundary cases that needs a better definition. The best report is often not the one with the highest value. It is the one that lets a skeptical reader see what was measured, what remains uncertain, and what the team did next.
Inter-annotator agreement is valuable because it turns a vague concern, “Are people applying this scheme in the same way?”, into evidence that a team can inspect. Used well, it catches brittle guidelines, weak calibration, unrepresentative assumptions, and tasks that were falsely presented as objective. Used lazily, it produces a decimal that looks stern and accomplishes very little. The difference is not statistical sophistication. It is whether the team treats agreement as the start of an inquiry.
