Probability, Statistics, Hypothesis Testing and Experiment Design
2.1 Foundations of Probability and Statistics
This section establishes three foundations: how to describe randomness, why a sample can represent a population, and why the sample mean can be approximated by a normal distribution. These ideas underpin every hypothesis test that follows.
Random Variables and Probability Distributions
- Random variable a quantity whose value depends on chance, such as the number of orders in a day or the amount one customer spends.
- Discrete values are countable points, such as the number of items in an order, the number of defect-free products or the number of clicks.
- Continuous values lie on an interval, such as money, duration or temperature.
A probability distribution is a probability map of possible values; it answers how common each outcome is. Before analysing data, ask: which type of quantity am I studying?
The Normal Distribution: Shape and Parameters
Many management measures are common near the centre and rare in the tails, so the normal distribution is the most common approximation.The mean positions the curve horizontally; the standard deviation controls how narrow or wide it is. The demonstration below lets you drag both parameters and see the shape change directly.
Standardisation, the Empirical Rule and Excel Functions
| Key idea | Meaning | Excel function |
|---|---|---|
| Standardisation | Convert a value into the number of standard deviations it lies from the mean | =STANDARDIZE(x, mean, standard_deviation) |
| Left-tail probability | Share of observations below a specified value | =NORM.DIST(x, mean, standard_deviation, TRUE) |
| Inverse calculation | Find the value associated with a specified probability | =NORM.INV(probability, mean, standard_deviation) |
| Empirical rule | About 68% lie within one standard deviation, about 95% within two and about 99.7% within three | — |
Populations, Samples and Random Sampling
- Population the complete set of objects of interest, such as the distribution of all order values.
- Sample the subset of data actually available, such as the 11,121 orders observed this month.
- Random sampling every object has the same probability of selection. Once sampling becomes “choosing the data,” every subsequent conclusion is invalid.
- Larger samples bring sample statistics closer to population parameters, but no increase in sample size can repair a biased sampling method.
Sampling Distributions and the Central Limit Theorem
If we repeatedly sample from the same population and calculate a sample mean each time, those means form a distribution called the sampling distribution of the sample mean. Its mean equals the population mean, and its standard deviation is called the standard error, equal to the population standard deviation divided by the square root of the sample size:
Central limit theorem for a sufficiently large sample, the distribution of the sample mean is approximately normal, regardless of the distribution of the underlying data. Even when the raw data are right-skewed, the sample mean can be approximated by a normal distribution; in practice, n around 30 is often sufficient. This is why confidence intervals and hypothesis tests work.
Confidence Intervals and Margin of Error
A point estimate gives one number; a confidence interval gives a range. Its width depends on the confidence level, standard deviation and sample size:margin of error = critical value × standard error. When the population standard deviation is unknown, substitute the sample standard deviation and replace the normal critical value with a t critical value.
Confidence Intervals in Excel
| Situation | Excel function | Method |
|---|---|---|
| Population standard deviation known | =CONFIDENCE.NORM(α, σ, n) | Use the normal critical value |
| Population standard deviation unknown | =CONFIDENCE.T(α, s, n) | Use the t critical value; this is the more common case |
| Interval notation | sample mean ± margin of error | For example, 102.84 ± 1.07 |
A confidence interval means that if we repeated sampling many times, about 95% of intervals would cover the true population mean. It does not mean “there is a 95% probability that the parameter lies in this particular interval”—the parameter is fixed; only the interval is random.
The Duality of Confidence Intervals and Hypothesis Tests
- If a 95% confidence interval excludes a value, a two-tailed test at that value would be rejected.
- The two tools are different statements of the same result; reporting the interval is more informative than reporting only a p-value.
- The interval also answers how large the effect might be and how uncertain it is—the information managers need.
2.2 The Four-Step Hypothesis-Testing Procedure
This section does one thing: turn “is this difference real?” into a repeatable process. Once you master the four steps, proportion tests, mean-difference tests and A/B tests all follow the same pattern.
Core Logic: Three Samples
A newly calibrated filling machine should place 200 mL in each bottle. Three samples produce three different sample means:
| Sample mean | Intuitive judgement | Reason |
|---|---|---|
| 170 mL | The machine is not calibrated | Too far from 200 to be ordinary random variation |
| 200.0 mL | The machine is operating correctly | Exactly at the target |
| 199.1 mL | Cannot tell; a test is needed | The difference is small, but cannot be judged by intuition alone |
The third case is what statistics is designed to resolve. The judgement must account for:sample size,the variability of the sample, and the significance level we are willing to accept.
Null and Alternative Hypotheses
Null hypothesis H₀
the proposition assumed true initially, usually “no change” or “no difference.”
Filling example: H₀: μ = 200
Alternative hypothesis HA
the proposition we hope the data will support.
Filling example: HA: μ ≠ 200
A test can reach only two conclusions:reject H₀ or fail to reject H₀. “Fail to reject” does not mean “H₀ has been proved”; it means the evidence is insufficient.
Rejection Regions and Significance Levels
The significance level α is the probability we accept of incorrectly rejecting H₀; 0.05 and 0.01 are common. If the sample statistic falls in the rejection region, reject the null hypothesis.
z and t Test Statistics
The test statistic measures how many standard errors separate the sample from the hypothesised value. Use z when the population standard deviation is known; use t with the sample standard deviation when it is unknown. The t distribution is heavier-tailed because it includes the uncertainty of estimating the standard deviation; degrees of freedom = n − 1.
As the sample grows, the t distribution approaches the normal distribution. Beyond n = 100, the critical values differ only slightly.
The Four-Step Hypothesis-Testing Procedure
- Formulate the hypotheses state the null H₀ and alternative HA; the null must contain an equality.
- Calculate the test statistic choose the correct formula: use t when the population standard deviation is unknown and z for proportions.
- Determine the critical value use the significance level and degrees of freedom to obtain the rejection region.
- Reach a conclusion if the statistic lies in the rejection region, reject H₀ and state the implication in business language: “therefore, we should do this.”
Complete the Four Steps for the Filling Example
The target is 200 mL. Ten bottles produce a sample mean of 199.1 mL and a sample standard deviation of 1.2 mL; α = 0.05. Choose an answer at each step and select Check.
One-Tailed Tests and Rules for Formulating the Null
Use a one-tailed test when only one direction matters; the rejection region lies at one end. A one-tailed critical value is smaller and makes rejection easier, so the direction must be specified before seeing the data; otherwise, you are effectively choosing the conclusion.
| Cannot serve as the null hypothesis | Can serve as the null hypothesis | Reason |
|---|---|---|
| ≠ | = | The null must contain an equality so that the rejection region has a clear boundary |
| > | ≥ | |
| < | ≤ |
Example: a manufacturer claims that mean male height exceeds 173 cm; write the null as μ ≤ 173 and the alternative as μ > 173.
Proportion Test: A Golf Course Example
Historically, 20% of customers were women. After a promotional campaign, a random sample of 400 customers contains 100 women, for a sample proportion of 25%. Is the five-percentage-point increase real or sampling variation?
| Item | Calculation | Result |
|---|---|---|
| Hypotheses | H₀: p ≤ 0.20, HA: p > 0.20 | One-tailed test |
| standard error | √(0.20 × 0.80 ÷ 400) | 0.02 |
| z statistic | (0.25 − 0.20) ÷ 0.02 | 2.50 |
| p-value | =1−NORM.S.DIST(2.50,TRUE) | 0.0062 |
| Conclusion | p < 0.05 | Reject H₀; the campaign is effective |
Business interpretation: there is sufficient evidence that the proportion of female customers exceeds 20%, so the campaign merits continuation.
Complete This Proportion Test Yourself
Using the data above, complete four fields: standard error, z statistic and one-tailed p-value, then choose the conclusion. Being in the correct order of magnitude counts as correct.
Difference in Two Means and Paired Tests
Whether two group means differ is one of the most common practical questions: average order value under old versus new approaches, dwell time for two user groups, or heights of male and female athletes. Before testing, ask:Are the two groups independent, or are these two observations of the same objects?
| Situation | Appropriate method | Key idea |
|---|---|---|
| Independent groups with similar spread | Two-sample t-test with equal variances | Degrees of freedom = n₁ + n₂ − 2 |
| Independent groups with clearly different spread | Welch two-sample t-test with unequal variances | Degrees of freedom require an adjustment; Excel calculates it automatically |
| Same objects observed before and after | Paired t-test | Tests the mean of the differences; observations must be matched one to one |
Type I and Type II Errors and Statistical Power
“Not significant” has two possible explanations: no real difference, or too small a sample. This is why an A/B test must begin with a sample-size calculation—if the sample is too small, even a completed experiment cannot support a conclusion.
Hypothesis-Testing Summary
- Four steps: formulate hypotheses, calculate the statistic, determine the critical value and reach a conclusion.
- Use t tests for means, z tests for proportions, and determine whether groups are paired before comparing them.
- Reject the null only when p < α; a p-value is not “the probability that the conclusion is correct.”
- Significance is not the same as importance; examine effect size and business meaning, and report the effect, confidence interval and sample size together.
Which Test Fits Each Situation?
The following five statements describe real business questions. Choose the most appropriate test for each.
2.3 AI-Assisted Testing and Experiment Design
AI plays two roles in this section: helping you write a complete experiment plan, and providing a “professional-looking” interpretation once results exist. The latter must be verified item by item.
Designing an Experiment with AI: One Prompt, Five Questions
Prompt template
I want to test whether [change] improves [metric], in the context of [setting and constraints].
Please provide: 1) the null and alternative hypotheses; 2) the appropriate test and rationale; 3) an approach for estimating sample size; 4) the assumptions to check; and 5) the three most common mistakes when drawing conclusions.
| Question | Requirement | Common error |
|---|---|---|
| How is the metric defined? | Provide the numerator, denominator and measurement window | Saying only “click-through rate” without defining the metric |
| How large a sample is needed? | Provide the calculation basis and minimum sample size | Claiming that “a few hundred observations should be enough” |
| How are users assigned? | Use random assignment and state the allocation ratio | Assigning by time sequence, which introduces temporal bias |
| How long should the experiment run? | Cover a complete business cycle | Running for only two days and ignoring weekday/weekend effects |
| When should the experiment stop? | Agree on a stopping rule in advance | Stopping when the result becomes significant, which inflates false positives |
Ask AI to Interpret Test Results, Then Verify Every Claim
Paste the Excel t statistic, p-value and the two group means into AI. Ask for a business-language interpretation and an actionable recommendation; then ask what assumptions the conclusion relies on and have AI list its three most likely errors. Finally,verify every item in Excel.
| Your verification action | Common AI error |
|---|---|
| Recalculate the test statistic and p-value with Excel Analysis ToolPak | Choosing the wrong test type, such as two-sample instead of paired |
| Check whether the two sample sizes and allocation ratio are balanced | Ignoring sample size and interpreting non-significance as no difference |
| Confirm one-tailed versus two-tailed | Mixing one- and two-tailed tests, or choosing the direction after seeing the data |
| Judge whether the effect size is large enough to justify action | Calling statistical significance a large business improvement |
| Check assumptions and experiment duration | Ignoring variance assumptions or the novelty effect |
Find the Problems in This AI-Generated Experiment Plan
Below is an A/B experiment plan generated by AI (a teaching example). Select a sentence to mark it as “problematic,” then select Check. The plan contains four problems.
Verification framework: metric definition, sample-size rationale, assignment method, experiment duration and stopping rule—one problem for each.
2.4 A/B Test Case: Comparing Headlines for Social-Media Content
This is the unit’s integrated case. A knowledge-focused social-media account wants more people to open the same piece of content: the platform randomly shows two versions to two groups, with Group A receiving the original headline and Group B the new headline. Every number below is calculated in real time from real data embedded in the page.
Why Run A/B Tests? First Understand Core Web Metrics
Websites and apps record everything: who saw what and when, what they clicked, how long they stayed and whether they returned the next day. These records are summarised into several common metrics.
| Metric | How it is calculated | Question it answers |
|---|---|---|
| Impression | Number of times a page or piece of content is displayed | How many people saw it? |
| Click-through rate (CTR) | clicks ÷ impressions | Of those who saw it, how many were interested? |
| Conversion rate (CVR) | Users completing the target action ÷ users reaching that stage | Of those who were interested, how many acted? |
| Retention rate | Users returning on day or week N ÷ first-time users | Did users stay? |
Arranging these stages in sequence produces a funnel model: users enter through impressions and proceed stage by stage, with some loss at every layer. The funnel’s value is that it breaks “poor performance” into “which layer underperforms”.
Retention is often shown as a retention curve: the horizontal axis is days since first use, and the vertical axis is the share returning that day. Flatter curves indicate stronger user stickiness; once two curves separate, the gap often widens over time.
Why do these metrics require A/B tests instead of simply comparing headline numbers? Four reasons:
- Metrics contain sampling variation. CTR and retention are proportions and naturally vary from day to day; differences visible to the eye may be noise.
- The effect to detect is often small. A two- to three-percentage-point lift from a headline change is already good performance and cannot be distinguished reliably by intuition.
- Metrics can work against one another. CTR may rise while reading time and retention fall—optimising one metric alone can lead to poor decisions.
- Only a randomised control design can separate the change from time, audience, season and other factors. This is precisely the problem addressed by the hypothesis testing and experiment design introduced earlier in this unit.
Case Background and Two Metrics
- Metric 1: Click-through rate = users who click ÷ users exposed. It is a proportion, so use a proportion test.
- Metric 2: Average reading time after clicking is a continuous variable available only for users who clicked, so use a two-sample t-test.
- Data fields: user ID, group (A/B), clicked (0/1) and reading time (seconds). The classroom version has 400 rows (200 per group), with a 4,000-row version demonstrating the effect of sample size.
Conclusion and Four Pitfalls
The classroom version concludes that the new headline significantly improves CTR (15.5% → 24.5%, z = 2.25, one-tailed p = 0.0122), while reading time does not change significantly (95.0 → 92.7 seconds, t = −0.314, p = 0.754). The business recommendation is to retain the new headline while continuing to monitor reading time.
| Pitfall | Meaning | Action |
|---|---|---|
| Significant ≠ important | A statistically real difference may not justify a costly business change | Assess both lift and cost before launching |
| Sample-ratio mismatch (SRM) | The allocation ratio deviates sharply from the design, indicating a problem with the experiment | Check assignment first; stop the experiment if the imbalance is severe |
| Multiple comparisons | When many metrics are tested simultaneously, at least one may appear “significant” | Specify primary metrics in advance and treat the rest as supporting evidence |
| Novelty effect | The novelty of the new version fades over time | Cover a full cycle and extend observation when necessary |
Write the Experiment Conclusion: Which Statements Are Correct?
The four statements below are common ways of reporting the result. Judge each as “sound” or “problematic,” then select Check.
2.5 Summary and Assignment
How to Choose a Test
| Question | Data type | Appropriate method |
|---|---|---|
| Does one proportion reach a specified level? | 0/1 data | One-sample proportion test |
| Do the proportions of two groups differ? | 0/1 data in two groups | Two-sample proportion test |
| Does one group mean reach a specified level? | Continuous data | One-sample t-test |
| Do two group means differ? | Independent continuous data in two groups | Two-sample t-test (first assess variance similarity) |
| Did the same objects change from before to after? | Paired continuous data | Paired t-test |
- Sampling distributions and the central limit theorem explain why the sample mean can be approximated by a normal distribution—the shared foundation of confidence intervals and tests.
- Confidence intervals and hypothesis tests are two sides of the same result; reporting an interval conveys more information than a p-value alone.
- The four-step procedure makes testing repeatable; when comparing groups, first determine whether the data are independent or paired.
- Three reminders for A/B tests: significance is not importance, calculate sample size in advance, and specify the stopping rule beforehand.
Appendix: Interactive Demonstrations and Classroom Exercises
| Location | Type | Meaning |
|---|---|---|
| 2.1 | Demonstration | Normal-distribution workbench: μ, σ, standard score and left-tail probability |
| 2.1 | Demonstration | Central limit theorem: repeated sampling from 11,121 orders |
| 2.1 | Demonstration | Confidence-interval workbench: repeated sampling and coverage |
| 2.2 | Demonstration | Difference in means: heights of 771 male and 667 female athletes |
| 2.4 | Demonstration | A/B workbench: 400 and 4,000 rows |
| 2.2 | Exercise | Four-step procedure for the filling case |
| 2.2 | Exercise | Proportion test for the golf-course case |
| 2.2 | Exercise | Choose a test for five scenarios |
| 2.3 | Exercise | Find four problems in an AI experiment plan |
| 2.4 | Exercise | Four ways to report an A/B conclusion |
Data sources: S1_online_orders.xlsx (11,121 order values), Class2.Data\Athletes.xlsx (heights of 771 male and 667 female athletes), and 单元2_AB测试数据_自媒体.xlsx (400-row classroom version and 4,000-row demonstration version). A/B data are simulated with a fixed random seed and are reproducible; every statistic on the page is calculated from the original data.
Unit Glossary (English–Chinese)
| English | 中文 |
|---|---|
| Random variable | 随机变量 |
| Normal distribution | 正态分布 |
| Sampling distribution | 抽样分布 |
| Central limit theorem | 中心极限定理 |
| Standard error | 标准误 |
| Confidence interval | 置信区间 |
| Null hypothesis | 原假设 |
| Significance level | 显著性水平 |
| p-value | p 值 |
| Type I error | 第一类错误 |
| Statistical power | 检验功效 |