Type I and Type II Errors, Power Analysis

Every hypothesis test can result in one of four outcomes depending on the true state of the world and the decision made. When H₀ is actually false and we reject it: correct decision — a true positive. When H₀ is actually true and we fail to reject it: correct decision — a true negative. When H₀ is actually true but we reject it: Type I error (false positive) — we concluded there is an effect when there is none. The probability of a Type I error equals alpha (α), the significance level — setting α = 0.05 means a 5% chance of incorrectly rejecting a true null hypothesis. When H₀ is actually false but we fail to reject it: Type II error (false negative) — we missed a real effect. The probability of a Type II error is called beta (β). The relationship between alpha and beta is not one-to-one: making alpha smaller (more stringent) reduces Type I errors but increases Type II errors for a fixed sample size. The stakes determine which error is more costly. In medical diagnostic testing: a Type I error (false positive cancer diagnosis) causes unnecessary treatment, anxiety, and cost. A Type II error (false negative cancer diagnosis) means a real cancer is missed — potentially fatal. Calibrate alpha based on the relative cost of each error type. In drug trials, the convention is α = 0.05 for efficacy (we can afford some Type I error) but the FDA requires very small alpha for safety signals (Type II errors are unacceptable when real side effects might go undetected).