Dimensionality reduction maps data into fewer variables, either by retaining selected original features or constructing a lower-dimensional representation. It can reduce storage and computation, support visualization, or regularize a downstream model, but it also discards information and can obscure interpretation.

Do not assume fewer dimensions will make a model more accurate. Define the goal first, fit every learned transformation on training data only, and compare the complete pipeline with an unreduced baseline under an evaluation design appropriate to the task.

Why reduce dimensions?

High dimensionality affects methods differently. In many metric spaces, observations become sparse and distance contrast can weaken as dimensions grow. Flexible models may also require more observations to estimate additional degrees of freedom. Redundant, noisy, or leakage-prone features can hurt, while genuinely informative features can help.

Feature selection keeps a subset of named inputs; feature extraction constructs new variables. If retaining original variables matters, start with the feature-selection guide. For broader workflow context, see feature engineering for machine learning.

PCA: variance-oriented linear projection

Principal Component Analysis (PCA) is a linear, unsupervised transformation. Scikit-learn centers the input and uses singular-value decomposition to project samples onto orthogonal directions ordered by the variance represented in the fitted data.

The loading vectors are orthogonal. In ordinary centered PCA, transformed scores have zero sample covariance apart from numerical and degeneracy considerations, but this is not a claim of statistical independence or a guarantee for new data. PCA does not use the target, and directions of greatest variance need not be most useful for prediction.

Explained variance is not task-relevant information. A low-variance direction may carry target signal, while a high-variance direction may be nuisance variation. Choose component count by examining cumulative explained variance together with held-out task performance, compute, and interpretability.

LDA: supervised separation

Linear Discriminant Analysis (LDA) is a supervised classifier that can also transform observations into a discriminant subspace. The classical model assumes class-conditional Gaussian distributions with a shared covariance matrix. Scikit-learn limits transformed dimensionality to at most min(n_features, n_classes - 1).

LDA seeks directions that separate labeled classes under its criterion, but it does not guarantee better classification. Estimate it within each training fold and compare the pipeline with regularized linear models, PCA, feature selection, and an unreduced baseline.

t-SNE: an exploratory neighborhood map

t-distributed Stochastic Neighbor Embedding (t-SNE) converts pairwise similarities into probability distributions in high- and low-dimensional spaces and optimizes a Kullback-Leibler divergence between them. Its non-convex objective can yield different embeddings from different initializations.

  • In scikit-learn, Barnes-Hut gradient computation is approximately O(N log N), while method="exact" is O(N²); wall-clock time also depends on data and hardware.
  • Cluster size, gaps, shapes, and distances between separated groups can change with preprocessing, metric, perplexity, initialization, and random seed.
  • Visible islands are not proof of classes, causal structure, or a biological mechanism. Repeat plausible settings and test substantive claims in the original feature space.

Scikit-learn’s load_digits dataset contains 1,797 8-by-8 handwritten digit images; it is not MNIST. Its t-SNE estimator provides fit_transform, not a reusable production transform for new observations.

UMAP: another neighborhood-based embedding

Uniform Manifold Approximation and Projection (UMAP) constructs a weighted nearest-neighbor graph and optimizes a low-dimensional representation. It is not simply t-SNE’s successor, and neither method dominates every dataset or goal.

n_neighbors, min_dist, metric, initialization, and random state materially affect a UMAP map. Larger n_neighbors incorporates broader neighborhoods; min_dist controls how tightly nearby points may be packed. Do not assume distances between clusters reproduce global distances in the original data.

Treat UMAP and t-SNE maps as exploratory views. Record preprocessing and parameters, set a seed, rerun plausible alternatives, and validate conclusions with original-space measurements.

Text and biological data need domain validation

Truncated SVD is commonly applicable to sparse term matrices because ordinary PCA centers data and may destroy sparsity. PCA can compress dense embeddings, but retained variance does not guarantee retained semantics. Evaluate retrieval or classification quality, calibration, subgroup behavior, latency, and storage on held-out data.

In gene-expression work, rows may represent samples or cells and columns genes. t-SNE and UMAP commonly visualize rows. Apparent islands do not establish cell types, pathways, or co-regulated genes. Define each point, address normalization and batch effects, and validate interpretations with suitable statistical and experimental evidence.

A validation-oriented chooser

  1. Define the goal: visualization, compression, denoising, and supervised prediction require different validation.
  2. Decide what must remain interpretable: selected features retain names; PCA components are combinations and are not automatically meaningful.
  3. Plan for new observations: choose a method with a defined out-of-sample transform when production inference requires one.
  4. Control target access: fit supervised reducers only inside training folds.
  5. Test stability: vary seeds and key hyperparameters; quantify trustworthiness or downstream performance rather than choosing the prettiest map.

A reducer is one component of an evaluated pipeline, not a universal remedy for overfitting. The bias-variance trade-off guide explains why reducing complexity can help or harm depending on signal, sample size, and model assumptions.

Sources

Fact-check record

Reviewed September 4, 2026. Automatic accuracy promises and arbitrary retained-variance targets were removed. PCA, LDA, t-SNE, UMAP, leakage, stability, out-of-sample use, and domain interpretation were corrected and qualified.