Feature engineering

Typical feature engineering operations include encoding categorical variables (one-hot, ordinal, or target encoding for high-cardinality columns), scaling numeric columns (standardisation, min-max scaling), non-linear transforms (log, Box-Cox, Yeo-Johnson, quantile), discretisation into bins, polynomial and interaction terms, date and time decomposition (day of week, hour, holiday flags), aggregations over windows or groups (counts, sums, rolling means per customer), and text representations such as bag-of-words or TF-IDF. scikit-learn's preprocessing module and ColumnTransformer implement most of these.

The main correctness risk is leakage. Every transform that learns statistics from data (a scaler's mean, a target encoder's category means, an imputer's median) must be fitted on training data only and then applied unchanged to validation, test and production data; scikit-learn recommends wrapping preprocessing and model in a Pipeline so that cross-validation refits the transforms inside each fold. Aggregate features must be computed only from information available at prediction time, and the same code path must run at training and serving time to avoid training-serving skew, which Google's production ML guidance lists as a primary monitoring target. Feature stores exist largely to share one definition across both paths.

Domingos (2012) summarised the empirical folk wisdom that the features used are often the most important factor in whether a project succeeds. Deep learning shifted much of this work into the model: convolutional and transformer networks learn representations directly from pixels or tokens, which is the core argument of representation learning (Goodfellow et al., ch. 15). For tabular data, however, gradient-boosted trees on engineered features remain highly competitive, and even deep pipelines still depend on engineering choices such as tokenisation, normalisation and data cleaning.