A/B Testing & Capstone Demo
This lesson covers A/B testing with PostHog and preparing for the capstone demo.
Like a restaurant testing two dishes with diners before adding to the menu — an A/B test tries two versions on real users before deciding what to ship.
- p-value
- The probability the test result was due to chance. p < 0.05 means less than a 5% chance of a false positive.
- Peeking problem
- Stopping a test early once p < 0.05 — inflates false positives.
- MDE
- Minimum Detectable Effect — the smallest lift the test is able to detect.
- Novelty effect
- A heightened response to a new change that fades over time — not necessarily a real improvement.
