In this section you'll continue to deepen your knowledge of hypothesis testing and t-tests by examining the concept of power; an idea closely related to type II errors. With that, you'll see how the rate of type I errors, power, sample size and effect size are intrinsically related to one another. From there, you'll investigate issues surrounding running multiple A/B testing experiments, which will lead you naturally into the concept of ANOVA - Analysis of Variance which allows you to effectively conduct multiple A/B tests simultaneously, testing for the influence of multiple factors all at once.
You will be able to:
- Define statistical power
- Define ANOVA
- Understand where ANOVA is used
Statistical power is equal to
After an initial exploration of statistical power, you'll take a look at Welch's t-test. This is an adaptation of the unpaired student's t-test you've seen previously which allows for different sample sizes or different variances between the two groups.
From there, you'll look at some of the issues that arise when trying to perform multiple comparisons - from the risks of spurious correlations to the importance of corrections such as the Bonferroni Correction to deal with the cumulative risks of type I errors inherent in multiple comparisons.
Finally, you'll take a look at the more generalized procedure for conducting multiple comparisons: Analysis of Variance or ANOVA. You'll see that ANOVA of only two groups is statistically equivalent to a two sided t-test. That said, ANOVA fully supports comparing multiple factors simultaneously.
Without a good understanding of experimental design, it's easy to end up drawing false conclusions. In this section, you'll cover a range of tools and techniques to deepen your understanding of hypothesis testing and ensure that you design experiments rigorously and interpret them thoughtfully.