Feature engineering turns information available at prediction time into variables a model can use. It can encode domain structure, improve efficiency, and make constraints explicit, but it does not guarantee better predictions. Every feature is a hypothesis to test against a reproducible baseline.

Begin with the prediction event

  1. Define the entity, target, prediction timestamp, horizon, and decision.
  2. Record when each candidate input becomes available. Exclude future values and target proxies.
  3. Reserve the final test set, using time-, entity-, group-, or geography-aware splits where needed.
  4. Fit imputers, encoders, scalers, selectors, and the estimator on training folds only.
  5. Add one defensible feature family at a time and run ablations under the same protocol.

Transforming the complete dataset before splitting leaks information. Pipelines help keep learned transformations inside each training fold. See feature selection methods and missing-data handling.

Numerical features

Ratios, differences, interactions, and domain aggregates may expose useful structure. Define units, guard against division by zero, and ensure every component exists at inference time. Linear models do not require predictors to be normally distributed. A logarithm can help with multiplicative relationships or scale compression, but log(x) requires positive values; log1p(x) accepts values greater than -1. Test transformations rather than applying them from histogram shape alone.

Scaling is important for many distance-based, regularized, and gradient-based estimators, but tree splits are generally invariant to monotonic scaling. Fit scaling statistics only on training data.

Categorical features

One-hot encoding is a dependable baseline for moderate cardinality; production pipelines must define unknown-category behavior. Ordinal encoding is appropriate only when order is real or the estimator can safely treat codes as labels rather than numeric distance.

Target encoding is supervised and especially leakage-prone. Compute it with cross-fitting inside training data, then fit the mapping on all training data for validation/test transformation. Apply smoothing and a documented fallback for unseen categories.

Time and event features

Calendar parts, lags, rolling summaries, and “time since” values must be computed as of the prediction timestamp. A seven-day rolling mean for a prediction at noon cannot include events recorded later that day. Use event time, ingestion time, cutoff rules, and late-arrival policy to reproduce historical values.

Respect temporal ordering in evaluation. Random row splits can let future observations or the same entity appear on both sides. For forecasting context, see time-series evaluation techniques.

Text and automated feature generation

Token counts, TF–IDF, embeddings, and model-derived attributes can be useful, but their vocabulary, weights, or fitted models belong inside the evaluation pipeline. Automated feature tools expand the search space; they do not remove the need for cutoff times, validation, resource controls, or domain review.

Evaluate features as system changes

  • Compare against a simple baseline on identical folds and seeds.
  • Report uncertainty or variation across folds/runs, not a single uplift.
  • Measure serving latency, freshness, compute, storage, and failure modes.
  • Check subgroup behavior, prohibited uses, and privacy constraints.
  • Reject features whose small offline gain is unstable or operationally costly.

Feature importance does not establish causality. Correlated predictors can share or obscure importance, and post-hoc explanations inherit model and data limitations. Spurious-correlation diagnostics can help challenge attractive but fragile signals.

Feature contracts for production

  • Document owner, definition, type, unit, source, availability time, freshness, range, missingness, and permitted use.
  • Version transformation code and fitted artifacts with the model and training snapshot.
  • Test offline/online equality on identical raw examples.
  • Monitor schema, missingness, unseen categories, distribution shift, latency, and downstream quality.
  • Define fallbacks for stale features and rollback for faulty transformations.

Frequently asked questions

Should I generate as many features as possible?

No. Broad searches increase compute and selection bias. Start with documented hypotheses and keep the test set untouched.

Does feature selection prevent overfitting?

No. It may reduce cost or variance, but can also discard signal or overfit validation. Treat the selector as a tuned pipeline step.

How do I know a feature is production-ready?

Confirm point-in-time availability, stable evaluation value, train/serve parity, acceptable cost, monitoring, ownership, and a fallback.