useR! 2016 Tutorial: Machine Learning Algorithmic Deep Dive

This tutorial contains training modules for six popular supervised machine learning methods:

• Classification and Regression Trees (CART)
• Random Forests (RF)
• Gradient Boosting Machines (GBM)
• Generalized Linear Models (GLM)
• Deep Neural Networks (DNN)
• Stacking / Super Learner (SL)

Here are some practical, related topics we will cover for each algorithm:

• Dimensionality Issues
• Sparsity
• Normalizaion
• Categorical Data
• Missing Data
• Class Imbalance
• Overfitting
• Software
• Scalability

Instructions for how to install the neccessary software for this tutorial is available here. Data for the tutorial can be downloaded by running ./data/get-data.sh (requires wget).

Certain algorithms don't scale well when there are millions of features. For example, decision trees require computing some sort of metric (to determine the splits) on all the feature values (or some fraction of the values as in Random Forest and Stochastic GBM). Therefore, computation time is linear in the number of features. Other algorithms, such as GLM, scale much better to high-dimensional (n << p) and wide data with appropriate regularization (e.g. Lasso, Elastic Net, Ridge).

Algorithms can deal with data sparsity (where many of the feature values are zero) in different ways. In some algorithms there are ways to speed up the computations if sparsity is present, so it's good to know if these shortcuts are available.

Some algorithms such as Deep Neural Nets and GLMs require that data be normalized for effective interpretation of the models. Tree-based algorithms (Decision Trees, Random Forest, Gradient Boosting Machines) do not require normalization. Tree-based methods only use information about whether a value is greater than or less than a certain value (e.g. x > 7 vs. x ≤ 7), the values themselves do not matter.

Algorithms handle categorical data differently. Some algorithms such as GLM and Deep Neural Nets require that a categorical variable be expanded into a set of indicator variables, prior to training. With tree-based methods and software that supports it, there are ways to get around this requirement, which allows the algorithm to handle the categorical features directly. It is important to be cognizant of the cardinality of your categorical features before training, as additional pre-processing (collapsing categories, etc) may be beneficial with high-cardinality features.

Assuming the features are missing completely at random, there are a number of ways of handling missing data:

1. Discard observations with any missing values.
2. Rely on the learning algorithm to deal with missing values in its training phase.
3. Impute all missing values before training.

For most learning methods, the imputation approach (3) is necessary. The simplest tactic is to impute the missing value with the mean or median of the nonmissing values for that feature. If the features have at least some moderate degree of dependence, one can do better by estimating a predictive model for each feature given the other features and then imputing each missing value by its prediction from the model.

Some software packages handle missing data automatically, although many don't, so it's important to be aware if any pre-processing is required by the user.

Algorithms that optimize a metric such as accuracy may fail to perform well on training sets that contain a significant degree of class imbalance. Certain algorithms, such as GBM, allow the user to optimize a performance metric of choice, which is useful when you have a highly imbalanced training set.

It is always good to pay attention to the potential of overfitting, but certain algorithms and certain implementations are more prone to this issue. For example, when using Deep Neural Nets and Gradient Boosting Machines, it's always a good idea to check for overfitting.

For each algorithm, we will provide examples of open source R packages that implement the algorithm. All implementations are different, so we will provide information on how each of the implementations differ.

We will address scalability issues inherent to the algorithm and discuss algorithmic or technological solutions to scalability concerns for "big data."

Where to learn more?

• An Introduction to Statistical Learning with Applications in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani
• The Elements of Statistical Learning by Trevor Hastie, Rob Tibshirani and Jerome Friedman
• Practical Data Science with R by John Mount and Nina Zumel
• Applied Predictive Modeling by Max Kuhn and Kjell Johnson
• 15 hours of expert videos by Trevor Hastie and Rob Tibshirani