Fact-check note: Reviewed September 4, 2026. This guide treats cleaning as controlled, documented handling of dataβ€”not a guarantee of flawless truth.

Data cleaning detects, investigates, and handles data that does not meet the requirements of a defined use. A value that looks unusual can be valid; a value that passes a format check can still be wrong. Preserve raw inputs, define acceptance criteria before changing records, and report residual limitations.

1. Define a contract and profile against it

Document field meaning, type, units, grain, permitted values, null policy, key scope, time zone, freshness, ownership, and change process. Profile types, nulls, ranges, uniqueness, distributions, patterns, and relationships by source, time, population, and pipeline stage. Profiling identifies review candidates; it does not establish correctness or representativeness.

The package formerly called pandas-profiling is now ydata-profiling. Generated reports can expose sensitive values and be costly on large data, so control access and sample deliberately.

2. Validate structure, semantics, and reconciliation

  • Parse against the source contract; a display regex is not global phone or email validity.
  • Define units, time zone, boundaries, and exceptions. A future order date may be valid for a scheduled order.
  • Test uniqueness only for a documented key and grain.
  • Verify approved relationships and reconcile counts, amounts, and control totals across pipeline boundaries.
  • Use regex only for limited syntax; prefer parsers, checksums, and authoritative references where appropriate.

3. Make transformations attributable

Record the input asset and version, rule or code version, parameters, execution identity and time, output asset, reason, changed and rejected counts, approvals, exceptions, and validation results. Preserve a protected source snapshot when required. Logs, Git, and dbt artifacts can contribute evidence, but none automatically forms a complete regulatory audit trail. Do not commit secrets, personal data, or large production extracts to Git merely to obtain history.

4. Analyze missingness and uncertainty

  • MCAR: missingness is independent of observed and unobserved data under the analysis model.
  • MAR: conditional on included observed information, missingness does not depend on the unobserved value. This is an assumption, not something observed data can prove.
  • MNAR: missingness can depend on the missing value after conditioning on observed information; analysis usually needs an explicit model and sensitivity analysis.

Choose complete-case analysis, weighting, model-based methods, imputation, or an explicit unknown category from the estimand, design, assumptions, and downstream use. Fit preprocessing only on the permitted training or analysis population to avoid leakage. Scikit-learn documents IterativeImputer as experimental; it returns one imputation by default and is not a drop-in equivalent to every MICE analysis.

5. Monitor critical quality indicators

For every check, define the quality dimension, population, threshold, window, owner, severity, response, exception path, and downstream consequence. A value beyond two standard deviations is not automatically anomalous in skewed, seasonal, autocorrelated, or shifting data. Detectors produce review signals and need false-positive, false-negative, and drift monitoring.

6. Standardize without hiding exceptions

Use versioned, tested, idempotent or compensatable transformations. At each boundary choose deliberately whether a failed check blocks, quarantines, warns, or continues. Orchestrators schedule and observe work, but retry, versioning, lineage, and replay behavior depend on implementation. Test partial failure and reconciliation.

7. Publish evidence and improve under change control

Publish provenance, contract version, checks, changed and rejected counts, reconciliations, uncertainty, known limitations, and approved uses. For machine learning, also test representativeness, label quality, leakage, subgroup behavior, and drift. Measure benefits against a baseline, such as fewer reconciliation breaks or lower review load; do not promise predictive accuracy or business returns without evidence and an attribution design.

Related guides: data strategy framework, data science project management, and MLOps best practices.