Data cleaning is the disciplined process of detecting, diagnosing, and addressing errors, inconsistencies, duplicates, missing values, and suspicious records while preserving what the original data was actually trying to say. The aim is not to make a dataset look neat, smooth, or reassuringly average. It is to make the dataset’s claims trustworthy enough for the model, analysis, or decision that will rely on them.
That distinction sounds fussy until a team starts deleting values. A blank may mean a broken sensor, an optional question, a privacy choice, or a value someone declined to report. A giant number may be a misplaced decimal, a unit mismatch, fraud, or the most informative observation in the file. Two nearly identical customer records may be redundant, or they may be two people who share an address and the bad luck of common names. None of these records arrives with a little tag saying “please replace me with the median.”
Cleaning is therefore closer to evidence handling than housekeeping. It begins with a suspicion, works backward toward a cause, and leaves a record of what changed and why. The word “clean” has unfortunately encouraged a kind of data minimalism, where anything awkward is treated as debris. A model trained on that dataset may be beautifully calibrated for a simplified version of reality, which is a lovely place to visit and an unhelpful place to deploy software.
Clean Is Not a Synonym for Uniform
A dataset can be consistent, complete, and still wrong for its purpose. A date formatted as 2026-03-07 may satisfy every syntactic rule while referring to the wrong event. A product category may be spelled consistently across thousands of rows and still reflect an outdated taxonomy. Conversely, a record with an unusual value may be entirely accurate. Data quality is not a property that floats above a dataset like a hygiene rating. It is fitness for a particular use.
That is why data preprocessing is a larger category than cleaning. Preprocessing also includes preparing usable inputs through scaling, encoding, feature construction, or dimensionality reduction. Cleaning comes earlier in the chain of thought: before deciding how to represent a value for a model, a team has to decide whether it is the right value, what it means, and whether its oddness contains information.
The classic database literature describes cleaning as detecting and repairing errors, but the word “repair” deserves some caution. A repair can be a confirmed correction from an authoritative source. It can also be a likely guess based on a rule, a peer group, or a statistical pattern. Those are different kinds of action. Systems such as HoloClean make that uncertainty explicit by combining rules, external evidence, and statistical properties in probabilistic repairs (Rekatsinas et al., 2017). A likely repair is useful, but it is still a likelihood, not a recovered memory.
A good cleaning workflow keeps the original record available, marks its status, and makes the new value traceable. That feels bureaucratic right up until the model begins making a strange decision six months later and someone needs to work out whether the problem came from the model, the pipeline, the source system, or an energetic spreadsheet edit on a Friday afternoon.
Flags Are Not Findings
Data validation checks whether data conforms to rules. A date should parse; a country code should appear in the approved set; a start date should not be after an end date; a quantity should not be negative if negative quantities are impossible in the domain. These checks are indispensable, and they produce useful flags.
But a flag is not a diagnosis. Suppose a record lists an age of 999. It might be a keying error. It might be a sentinel code for “unknown.” It might be a convention inherited from a legacy application that nobody documented. A validation rule can tell you that 999 is implausible as an age. It cannot decide whether the correct action is to restore a source value, map the sentinel to missingness, quarantine the record, or preserve it until the source owner explains it.
This separation between error detection and error diagnosis is the part of cleaning that automation most often tries to rush past. Constraint-based methods detect values that cannot coexist under a known rule. Statistical methods identify observations that deviate from an expected pattern. Both are productive ways to find candidates, but neither creates an authoritative repair on its own (Chu et al., 2016). The right next move depends on source semantics.
The same applies to an integrity constraint, a statement about relationships the data is supposed to obey. If two records with the same postal code list incompatible regions, the constraint identifies a contradiction. It does not tell you whether the postal code, the region, or the assumption about the relationship is wrong. A rule can prove that a set of cells cannot all be correct. It cannot point at the guilty cell without some additional evidence. Databases, like group projects, are quite good at revealing that someone is mistaken and less good at identifying who.
The table is not a recipe card. It is a reminder that anomaly handling begins with an investigative question. The most defensible action is sometimes to preserve the record exactly as it is, along with a clear status indicating that it is uncertain or needs review.
Absence Has a History
Missing data is often treated as a formatting inconvenience: replace blanks, move on, enjoy the satisfying lack of nulls. That approach mistakes a condition for its cause. Missingness is itself a pattern, and the mechanism behind it determines whether a repair is safe.
Statisticians commonly distinguish among three broad mechanisms. Missing completely at random, usually written as MCAR, means the absence is unrelated to observed or unobserved values. A system outage that randomly drops a few readings is the familiar example. Missing at random, or MAR, means missingness is related to observed information. A follow-up survey might be absent more often for records from a certain region. Missing not at random, MNAR, means the likelihood that a value is missing is related to the unobserved value itself, such as a respondent declining to report a particularly sensitive number (Sharifnia et al., 2025).
Those names are not merely a statistics exam waiting to happen. They determine what a model can infer from the remaining data. Deleting incomplete rows can distort a dataset if the absence is concentrated in a meaningful subgroup. Filling every blank with an average can narrow the observed distribution and erase the fact that a value was unavailable. Imputation is an estimate, not a time machine. It can be exactly the right choice, but only when its assumptions are visible and proportionate to the task.
For an ML system, a missingness indicator can sometimes be useful. It preserves the fact that a value was absent while another field supplies an imputed value or an explicit placeholder. Yet even that needs care. If a model learns that a field is absent because of a temporary collection glitch, it may depend on a pattern that disappears after the pipeline is fixed. A missingness feature is a claim about the environment, just like any other feature. Treat it with the same suspicion you would apply to a surprisingly predictive ZIP code.
The Same Thing Can Arrive Twice
Duplicate records are not always literal copies. A person may appear under a nickname in one source and a legal name in another. A company can have a registered address and a shipping address. An event can be reported by two systems with slightly different timestamps. Entity resolution, also called record linkage or deduplication, is the task of identifying records that refer to the same real-world entity despite those differences.
This is where “remove duplicates” becomes a dangerous instruction. Exact matching is cheap and often useful. Near matching asks for a similarity rule, candidate blocking, thresholds, and an explanation of what evidence is sufficient to merge. A correct merge can consolidate fragmented history. A false merge can blend two distinct people or events into a polished but fictional golden record. That is not cleaning. It is administrative fan fiction.
The danger reaches model evaluation quickly. If a training set contains one version of a record and the test set contains a near duplicate, the model receives an illicit preview of its exam. This form of train-test leakage often looks less dramatic than a duplicated row, but it can make performance estimates deceptively generous. Deduplicate, or at least group related records, before splitting data for training and testing. The split should separate underlying entities, not merely lines in a CSV.
A merge also needs a survivor policy. Which spelling wins? Which source is authoritative for the address? Is the newest value preferable, or is the original source more reliable? Record those rules. Otherwise, a data pipeline can silently convert a reversible decision into a permanent fact, and subsequent users will inherit a value that looks native to the source even though it was assembled from several sources later.
Rare Is Not Wrong
An outlier is an observation that departs enough from an expected pattern to deserve scrutiny. The final phrase matters. An outlier score is not a delete button wearing a lab coat.
Outlier-detection methods make assumptions about normality. A statistics-based method may expect ordinary values to occupy high-probability regions. A distance-based method may treat a point far from its neighbors as unusual. A model-based method may learn what ordinary examples look like, then flag deviations. Each can be appropriate in the right setting, but each also bakes in a particular view of what “normal” means (Chandola et al., 2009).
Consider a sensor record that reads 250 when neighboring values sit near 25. It could be an extra zero. It could be a unit mismatch. It could be a genuine spike tied to a documented event. It could also be ordinary on its own but implausible only when paired with another field. The decision to alter it should follow source checks, unit checks, related data, and the cost of retaining or losing rare cases. A fraud detector that cleans away unusual transactions is not so much a detector as an accomplice with tidy spreadsheets.
When a value is genuine but influential, sensitivity analysis is often more honest than deletion. Run the relevant analysis or model with and without the case, document the effect, and decide whether the intended decision needs a robust method, a separate treatment, or additional collection. Data cleaning should reduce accidental distortion, not sand down the part of reality that makes a metric less photogenic.
Models Can Help Find Dirty Labels
For supervised systems, the target label is data too. A feature table can be perfectly formatted while labels say that a picture of a missile belongs to its broader parent category, a refund request is “praise,” or a medical finding has the wrong class. Label errors are especially expensive because they teach a model that a contradiction is part of the task.
Confident learning uses a model’s predicted probabilities alongside observed labels to estimate likely label-noise patterns and rank examples for inspection. Its useful contribution is not the promise of an automatic truth machine. It is a way to prioritize limited review time toward examples where the model and the dataset disagree in structured, informative ways (Northcutt et al., 2021).
That distinction matters because model disagreement can expose more than a wrong label. It can reveal ambiguous instructions, overlapping categories, source-specific artifacts, or a population shift the label schema never anticipated. Data annotation produces the learning signal; cleaning gives teams a method for revisiting it when the signal and the evidence no longer line up.
Model-aware cleaning can also help allocate effort. BoostClean and later systems evaluate cleaning choices by whether they improve performance on an intended downstream task, rather than assuming every repair has equal value (Krishnan et al., 2017). That is sensible prioritization. It does not mean an accuracy gain excuses a repair that destroys provenance, hides a disparity, or alters data beyond what the source supports. A model metric is a useful witness. It should not be the only judge.
A Repair Needs a Receipt
The best cleaning pipelines have a slightly annoying quality: they remember everything. They retain raw data, preserve a cleaned layer, record the rule or evidence behind each modification, version the code, and provide a way to reproduce or reverse an action. This is data provenance, the lineage that explains where a value came from and how it changed.
Provenance turns a cleaning decision from a silent mutation into an inspectable claim. A reviewer should be able to ask: Which source supplied this value? Was it standardized, inferred, merged, or manually corrected? Which rule ran? Who approved the exception? What confidence did the repair system assign? If no one can answer, the dataset may be clean enough to demo and too mysterious to maintain.
Documentation matters here because schemas age. A field called status might switch from a human-entered workflow state to an automated system state without changing its column name. Rules that were once correct can become elegant machinery for producing wrong data. In software-backed datasets, tools such as Sandgarden's Find the Gaps can help expose places where documentation has drifted from the code that defines current behavior. That will not repair a dataset on its own, but it can reveal why a once-reasonable cleaning rule has started making bad guesses.
The practical standard is not a mythical perfectly clean dataset. It is a dataset whose uncertainties, repairs, and remaining limitations are known well enough that people and models can use it without pretending it is something else. Cleaning succeeds when it preserves the trail from messy observation to justified decision. Everything after that, including a model’s confidence score, has a sturdier place to stand.
