Data cleaning in Python should be a documented transformation from an immutable source to a validated analytical dataset. The objective is not to make every value look tidy; it is to identify defects, preserve evidence, apply justified rules, and make the result reproducible.

Load data without erasing evidence

Keep the raw file unchanged, declare identifiers as strings when leading zeros matter, and verify required columns before transformation.

from pathlib import Path
import pandas as pd

source = Path("customer_data.csv")
required = {"customer_id", "signup_date", "total_spend", "customer_age", "country"}
raw = pd.read_csv(source, dtype={"customer_id": "string", "country": "string"})
missing = required.difference(raw.columns)
if missing:
    raise ValueError(f"Missing required columns: {sorted(missing)}")
df = raw.copy()

head(), info(), and describe() are reconnaissance, not proof of quality. Check grain, uniqueness, referential integrity, vocabulary, units, ranges, time zones, parsing failures, and distributions by relevant group.

Parse explicitly and retain failures

spend_text = df["total_spend"].astype("string").str.strip()
df["total_spend"] = pd.to_numeric(
    spend_text.str.replace(r"[$,]", "", regex=True), errors="coerce"
)
df["signup_date"] = pd.to_datetime(df["signup_date"], errors="coerce", utc=True)
bad_spend = spend_text.notna() & df["total_spend"].isna()
rejected_spend = raw.loc[bad_spend].assign(reason="invalid_total_spend")

Coercion is safe only when failures remain available for review. A negative spend may be an error, refund, or adjustment. An implausible age should be quarantined and investigated, not silently replaced.

Treat missingness as information

There is no universal percentage at which deletion becomes safe. Determine what a null means, how it was produced, which groups it affects, and whether the field is required. Simple imputation is a baseline, not recovered truth.

df["customer_age_was_missing"] = df["customer_age"].isna()
median_age = df["customer_age"].median(skipna=True)
df["customer_age"] = df["customer_age"].fillna(median_age)

For predictive modeling, fit the median and every other learned transformation on each training fold only. Forward fill requires explicit entity ordering and a defensible persistence assumption; backward fill uses future information and is usually unsuitable for real-time prediction.

Normalize categories with governed mappings

country_map = {"usa": "United States", "u.s.a.": "United States",
               "united states": "United States", "canada": "Canada",
               "mexico": "Mexico"}
normalized = df["country"].str.strip().str.casefold()
df["country_clean"] = normalized.map(country_map)
df["country_unmapped"] = normalized.notna() & df["country_clean"].isna()

Retain the raw value and mapping-table version. Flag unfamiliar categories rather than guessing.

Resolve duplicates and outliers by domain rules

drop_duplicates() removes identical rows; it does not determine whether different records represent the same entity. Define the grain and key, inspect conflicts, and record survivor or merge decisions. Statistical rarity is not invalidity: investigate source records, units, sampling, and business events before capping or excluding an outlier.

Validate and publish an audit

assert df["customer_id"].notna().all()
assert df["customer_id"].is_unique
assert df["country_unmapped"].sum() == 0

Record input and output checksums, code and dependency versions, row and null counts, rejection counts, rule versions, and execution time. Replace these illustrative assertions with the actual data contract.

  • Preserve raw inputs and access controls.
  • Define schema, keys, units, vocabulary, and time rules.
  • Retain rejected rows with reason codes.
  • Fit learned preprocessing inside training folds.
  • Version transformations and test critical constraints.

See data-cleaning best practices, then continue with feature engineering. Clean data strengthens—but does not guarantee—sound decision-making.

Originally published August 12, 2025; technically reviewed and substantially updated September 4, 2026.