Missing data is not only a blank-cell problem. A value may be absent because it was never collected, was not applicable, failed validation, arrived late, was censored, or was withheld. The appropriate response depends on the analysis goal, collection process, timing, and assumptions about why values are missing.

Profile missingness before changing the data

Preserve the raw data and missing-value codes. Confirm that placeholders such as empty strings, sentinel numbers, and “unknown” labels are interpreted correctly. Report missingness by variable, row, time period, source, and relevant group, then compare complete and incomplete cases on observed fields.

missing_by_column = df.isna().mean().sort_values(ascending=False)
missing_by_row = df.isna().sum(axis=1)

print(missing_by_column)
print(missing_by_row.describe())

A sparse feature is not necessarily useless. Absence may carry information or affect a small but important group. Review semantics, collection changes, leakage, fairness, and downstream use before dropping it. See data cleaning best practices for a broader audit workflow.

Make missing-data assumptions explicit

Rubin’s framework describes assumptions about the probability that data are missing:

  • Missing completely at random (MCAR): missingness does not depend on observed or missing values in the analysis. This is a strong assumption.
  • Missing at random (MAR): after conditioning on observed information included in the model, missingness does not additionally depend on the unseen value. “Random” does not mean unrelated to everything.
  • Missing not at random (MNAR): the MAR condition does not hold; missingness still depends on unobserved information after conditioning on observed data.

These are assumptions about a data-generating process, not labels usually proven from the incomplete table alone. Use domain knowledge, collection metadata, time patterns, and observed comparisons. If plausible MNAR mechanisms could change the conclusion, perform sensitivity analyses.

Separate deletion decisions

Complete-case or listwise deletion analyzes only rows complete for every variable required by that analysis. It changes the analyzed population and reduces sample size. Whether it biases an estimate depends on the missingness process, variables, and model.

Pairwise deletion does not mean removing columns. It calculates each statistic using cases complete for that variable pair, producing different sample bases and potentially an invalid covariance matrix. Dropping a feature is a separate design choice.

There is no universal percentage below which deletion is safe. Even a small proportion can bias a consequential estimate if missingness is systematic; a larger proportion may be manageable under a justified model. Quantify affected observations and groups, state assumptions, and compare reasonable alternatives.

Choose an approach that matches the goal

ApproachPotential useMain cautions
Complete-case analysisA transparent primary or sensitivity analysis when assumptions are defensibleChanged population, reduced precision, and possible bias
Constant, mean, or median imputationA simple predictive baselineAltered distributions and relationships; uncertainty is not represented
Missingness indicatorAllows a predictor to use observed absence patternsMay encode collection artifacts or sensitive group differences
K-nearest-neighbor imputationUses relationships among nearby samplesScale, irrelevant features, density, distance, and neighbor choice matter
Iterative imputationModels each incomplete feature from othersMisspecification, cost, leakage, experimental API status, and single-imputation uncertainty
Multiple imputationRepresents uncertainty across several completed datasets for suitable inferenceRequires a compatible imputation model and correct pooling procedure

No method is uniformly best. For prediction, compare justified options within the same validation design using performance, calibration, subgroup behavior, stability, and cost. For inference, use methods and uncertainty procedures appropriate to the estimand.

Prevent leakage with training-only pipelines

Splitting after imputation lets information from held-out data influence preprocessing. Split first and fit learned transformations only on training folds. This principle also applies to scaling, encoding, feature selection, and resampling; see feature engineering for machine learning.

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = make_pipeline(
    SimpleImputer(strategy="median", add_indicator=True),
    LogisticRegression(max_iter=1_000),
)

model.fit(X_train, y_train)
test_score = model.score(X_test, y_test)

During cross-validation, the pipeline learns imputation values separately from each training fold. The indicator may help when absence is predictive, but validate its effect and consider whether it captures an unstable collection process.

Use KNN imputation carefully

KNNImputer fills each missing feature using neighboring samples that have a value for that feature. Scikit-learn’s default nan_euclidean distance compares features observed in both samples; neighbors need not be complete rows. The estimate is the mean of selected neighbors, optionally distance-weighted.

Results depend on feature scale, irrelevant variables, sample density, missingness, and n_neighbors. For mixed or differently scaled features, build appropriate column-wise preprocessing and avoid letting a large-unit feature dominate distance. Fit the entire preprocessing design on training data and compare it with simpler baselines.

Understand IterativeImputer and multiple imputation

Scikit-learn’s IterativeImputer models each incomplete feature as a function of other features in a round-robin sequence. In the reviewed documentation it remains experimental and must be enabled explicitly. Its default estimator is BayesianRidge, so it should not be described as automatically finding complex nonlinear relationships.

from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import IterativeImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

iterative_model = make_pipeline(
    IterativeImputer(random_state=42),
    LogisticRegression(max_iter=1_000),
)

iterative_model.fit(X_train, y_train)
iterative_test_score = iterative_model.score(X_test, y_test)

The imputer returns a single completed dataset by default. Multiple imputation requires repeated draws under a suitable model and a downstream pooling procedure that propagates uncertainty. Better reconstruction or prediction does not by itself make an inferential estimate unbiased.

Native missing-value handling is not a missing-data solution

Some estimators define routing rules for missing feature values. This removes the mechanical need to fill every cell, but it does not identify why data is missing, correct selection bias, prevent leakage, or guarantee valid inference. Verify behavior for the exact library version and estimator configuration.

Run sensitivity analyses

Pre-specify a primary approach when possible, then compare conclusions under credible alternatives. For prediction, keep the same untouched evaluation design and compare complete-case, simple-imputation, and justified model-based strategies. For inference, vary assumptions and imputation models in ways that reflect plausible missingness mechanisms.

  • Report records and groups included under each strategy.
  • Compare estimates, uncertainty intervals, calibration, and subgroup results.
  • Test time or source changes that may alter missingness.
  • Document where conclusions are stable and where they depend on assumptions.

Basic statistical concepts behind uncertainty and sensitivity are reviewed in basic statistics concepts.

Frequently asked questions

What percentage of missing data is acceptable to delete?

There is no universal threshold. Report missingness by variable and case, investigate collection, quantify affected groups, and compare conclusions under plausible alternatives.

Is imputation safer than deletion?

Not automatically. Both rely on assumptions and can bias an analysis. The choice depends on the estimand, prediction setting, missingness process, and uncertainty treatment.

Does an imputer restore the true value?

No. It supplies a model-based value or distribution conditional on assumptions and observed information. Preserve that uncertainty in interpretation.

Handling missing data well means documenting how values became absent, matching the method to the analytical goal, preventing leakage, and showing whether reasonable alternative assumptions change the result.