Feature Selection: Methods, Leakage, and Evaluation

Feature selection is a model-selection problem

Feature selection keeps a subset of the original variables. It can reduce measurement cost, simplify a model, or make a training pipeline more practical, but it does not automatically improve accuracy or prevent overfitting. The result depends on the data, estimator, scoring rule, validation design, and stability of the selected set.

Three broad families are useful: filters score features without repeatedly fitting the final predictor; wrappers evaluate subsets with a predictor; and embedded methods produce sparsity or feature importance during model fitting. Principal component analysis (PCA) belongs beside this taxonomy as feature extraction: it creates new linear components rather than selecting original variables.

Feature-reduction families and their main limitations
Family	What it does	Examples	Main caution
Filter	Scores or removes features without repeatedly fitting the final predictor	Variance threshold, F-tests, chi-square, mutual information	Univariate scores can miss redundancy and joint effects; assumptions and data types matter
Wrapper	Searches feature subsets using an estimator and validation score	Forward selection, backward selection, RFE, RFECV	Computationally expensive and prone to selection overfitting if evaluation is reused
Embedded	Uses sparsity or learned importance from model fitting, then applies a selection rule	L1-regularized models, tree models with `SelectFromModel`	Model- and hyperparameter-dependent; importance can be unstable or biased
Feature extraction	Creates a lower-dimensional representation	PCA	Components are not original features and high predictor variance need not be target-relevant

Start with the evaluation design

Define the prediction task and metric. Use a metric appropriate to the task, class balance, decision costs, and calibration needs—not a generic accuracy checklist.
Reserve final evaluation data. Keep an untouched test set when the data volume permits. For small datasets or extensive tuning, consider nested cross-validation.
Fit every learned preprocessing step inside resampling. Imputation, scaling, supervised feature scoring, thresholds, and subset search must learn from the training fold only. A pipeline is the safest default.
Compare against a no-selection baseline. Report predictive performance, uncertainty, feature count, runtime, and operational cost. Selection is useful only if it improves the outcome that matters.
Check stability. Repeat the procedure across folds, seeds, time windows, or bootstrap samples. Investigate features whose inclusion changes frequently, especially when predictors are correlated.
Evaluate once on held-out data. Do not choose the method or threshold using the same result later presented as the final unbiased score.

Filter methods: fast ranking with explicit assumptions

Univariate filter procedures evaluate one feature at a time. They are useful for screening very wide datasets, but they can select redundant variables and miss features that matter only jointly.

Variance threshold: removes features with little or no variation without using the target. A low-variance feature can still be useful, so the threshold is a domain decision.
F-tests: test linear dependence under their stated assumptions. They should not be described as general tests of all dependence.
Chi-square: can rank nonnegative features for classification. Scikit-learn requires nonnegative input; suitable counts or transformed values are common examples.
Mutual information: can detect nonlinear statistical dependence between one feature and the target. Estimates for continuous variables depend on the estimator and sample size, and pairwise MI does not reveal every multivariate interaction.

Tune the score function and number of retained features inside cross-validation. Combining several rankings does not become “robust” merely because more scores were used; define the combination rule and validate it.

Wrapper methods: estimator-specific subset search

Sequential feature selection and recursive feature elimination repeatedly fit an estimator while adding or removing features. This can align the subset with a particular model and metric, but the search is greedy or otherwise heuristic unless every subset is evaluated. It does not guarantee a globally optimal set.

RFE requires an estimator with usable coefficients or feature importances, or a custom importance getter. RFECV repeats RFE across cross-validation splits and chooses the feature count with the best mean configured score. That result is conditional on the estimator, split strategy, metric, step size, and hyperparameters.

Use wrappers only inside the training process. If RFECV or sequential selection determines the subset and an outer test score is needed, keep that test data untouched or use an outer validation loop.

Embedded methods: sparsity and learned importance

L1-regularized linear models can set some coefficients to zero, which supplies an embedded selection mechanism. Standardize features when their scales should be comparable, tune the penalty on training folds, and inspect selection stability. With correlated predictors, one variable may be retained while another similar variable is dropped; the selected support should not be presented as a causal discovery.

Tree ensembles expose feature-importance measures, but importance is not the same as selection. A separate threshold or meta-transformer defines which variables remain. Impurity-based importance can favor high-cardinality features. Held-out permutation importance can provide another view, although correlated features can distribute importance across substitutes.

Where PCA fits

PCA centers the predictor matrix and projects it onto orthogonal directions that explain decreasing amounts of predictor variance. It creates components rather than retaining named input features, so it is feature extraction. Scikit-learn's PCA centers input but does not scale each feature by default; scaling should be a deliberate choice based on units and meaning.

Explained variance is not predictive relevance. Select the number of components within the training pipeline and validate the downstream task. Use PCA when a compact linear representation is acceptable; avoid it when retaining original-variable meaning or measurement decisions is the objective.

A leakage-safe scikit-learn pattern

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("select", SelectKBest(score_func=f_classif)),
    ("model", LogisticRegression(max_iter=2000)),
])

search = GridSearchCV(
    pipe,
    param_grid={
        "select__k": [5, 10, 20, "all"],
        "model__C": [0.1, 1.0, 10.0],
    },
    scoring="roc_auc",
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    n_jobs=-1,
)

# X_test and y_test must not have influenced the grid or feature scores.
search.fit(X_train, y_train)
held_out_auc = search.score(X_test, y_test)

This is an illustration, not a universal recipe. Choose preprocessing, score function, split strategy, and metric for the data and decision. Time-dependent, grouped, imbalanced, or multimodal datasets usually need a different split design.

How to choose among the approaches

Use a filter as a computational screen when the feature space is very wide and its assumptions are appropriate.
Use a wrapper when the estimator-specific subset is worth repeated fitting and the sample size supports a separate outer evaluation.
Use an embedded method when sparsity or learned importance matches the estimator, while checking hyperparameter and resampling stability.
Use PCA when a compact component representation is acceptable and retaining original feature identity is not required.

Document the candidate features, exclusions, transformations, selection rule, resampling design, metric, retained subset, stability, and final held-out result. That record is more useful than claiming that one technique is universally best.