1 Feature Engineering (Max Kuhn)

Max Kuhn - Feature Engineering

To avoid data leakage in feature engineering, first split the entire dataset into training and holdout sets, and only do feature engineering on the training data. This also applies when you use mean encoding to transform a categorical feature to a number: split the data first and then compute the sample mean of the label, based on the training data only.

1.1 Introduction

  • Concepts

    • Overfitting

      • Overfitting is the situation where a model fits very well to the current data but fails when predicting new samples.

      • feature selection techniques can overfit to the predictors. This occurs when a variable appears relevant in the current data set but shows no real relationship with the outcome once new data are collected.

    • No free-lunch

      • The “No Free Lunch” Theorem (Wolpert 1996) is the idea that, without any specific knowledge of the problem or data at hand, no one predictive model can be said to be the best.

      • In practice, it is wise to try a number of disparate types of models to probe which ones will work well with your particular data set.

    • The Model versus the Modeling Process

      • The process of developing an effective model is both iterative and heuristic.

Modeling Process

  • Model Bias and Variance

  • Experience-Driven Modeling and Empirically Driven Modeling

    • It is common to have some conflict between experience-driven modeling and empirically driven modeling. Each approach has its advantages and disadvantages.

    • In practice, we have found that a combination of the two approaches works best as long as both sides see the value in the contrasting approaches.

    • The subject matter expert may have more confidence in a novel model feature if they feel that the methodology used to discover the feature is rigorous enough to avoid spurious results.

    • Also an empirical modeler might find benefit in an expert’s recommendations to initially whittle down a large number of predictors or at least to help prioritize them in the modeling process.

    • The process of feature engineering requires some level of expertise related to what is being modeled.

  • Big Data

    • Big data does not necessarily mean better data.

    • Overall, when encountering (or being offered) large amounts of data, one might think to ask:

      • What are you using it for? Does it solve some unmet need?

      • Will it get in the way?