Unit 3: Required

In this lesson, we use the Boston housing dataset (imported from scikit-learn) and the average weight of mammal bodies/brains (included). These datasets are appropriate for linear modeling based on their generally intuitive features.

After this lesson, students will be able to:

• Describe what error due to bias is and what error due to variance is
• Identify the bias-variance tradeoff
• Describe what overfitting and underfitting means in the context of model building
• Explain problems associated with over and underfitting
• Grasp why train, test split is necessary
• Explore kfolds, LOOCV, and three split methods

Before this lesson(s), students should already be able to:

• Read into data using the Pandas library
• Perform statistical exploration with Pandas
• Create simple data visualizations with matplotlib
• Define the basic parameters of sampling and experimental design

TOTAL (170 min)

• Bias and Variance Trade-off (35 min)
  • Bias? Variance? (10 min)
  • Exploring the Bias-Variance Tradeoff (15 min)
  • Brain and body weight mammal dataset (5 min)
  • Making a prediction (5 min)
• Making a prediction from a sample (15 min)
  • Let's try something completely different
• Balancing Bias and Variance (10 min)
• Train-test-split (50 min)
  • Evaluation procedure #1: Train and test on the entire dataset (do not do this)
  • Problems with training and testing on the same data
  • Evaluation procedure #2: Train/test split
  • Comparing test performance with a null baseline
• K-folds cross-validation (45 min)
  • Leave-one-out-cross-validation
  • Intro to cross validation with the Boston data
• Three way data split (15 min)
• Additional Resources

For more information on this topic, check out the following resources:

• Understanding the Bias-Variance Tradeoff
• University of Washington Machine Learning Course Slides
• An Intuitive Explanation of Overfitting