Experiments, hypothesis tests, and effect sizes

Prerequisites: Conditional probability, sampling, confidence intervals, and basic means.

Learning objectives

• Explain the role of random assignment.
• Interpret a p-value without reversing its condition.
• Separate statistical evidence, effect magnitude, and practical importance.

State the decision and outcome before collecting data

An experiment deliberately varies a treatment and measures an outcome. Random assignment makes treatment groups comparable in expectation by breaking systematic links between assignment and preexisting characteristics. It does not guarantee perfectly balanced groups in every finite experiment, and it does not automatically make participants representative of a broader population.

Suppose a learning platform compares two navigation designs. Define the experimental unit, such as an account, and assign each unit consistently to one design. Choose an outcome, such as successful completion of a specified task, a time window, and a handling rule for missing outcomes. Randomizing each page view could expose the same person to both designs and create dependence or interference that the analysis must address.

A test evaluates a specified null model

A null hypothesis might say the two designs have equal completion probabilities. A test statistic measures disagreement with that model. A p-value is the probability, assuming the null model and test assumptions, of obtaining a statistic at least as extreme as the observed one under the specified extremeness rule. It is not the probability that the null hypothesis is true, and it is not the probability that the result happened “by chance.”

For a simple illustration, a supposedly fair coin produces heads in all ten independent tosses. The probability of that exact sequence under fairness is 1/1024. For a two-sided test based on distance from five heads, both all-heads and all-tails are equally extreme, giving p=2/1024, about 0.00195. The extremeness definition changes which outcomes count; it must not be invented after seeing the result.

Error rates belong to procedures

A significance threshold alpha controls the test's false-positive probability under its assumptions. A small p-value does not give the probability that this particular rejection is wrong. Power is the probability of rejecting under a specified alternative, and depends on effect size, variation, sample size, and the test.

Repeatedly checking results and stopping at the first small p-value can inflate false positives unless the procedure accounts for sequential monitoring. Testing many outcomes and reporting only the smallest p-value creates a related selection problem.

Report effect sizes and uncertainty alongside test results. A tiny improvement can be statistically detectable in a large sample yet irrelevant to learners, while a meaningful effect can remain uncertain in a small sample. Predeclare the main comparison, inspect data quality, and discuss interference, attrition, and population limits. Statistical significance alone is not a product decision.

Practice problem

Design an experiment comparing two lesson layouts. Specify assignment unit, primary outcome, observation window, and a stopping rule. Explain how users who never finish should be handled without selecting only successful participants. Then interpret p=0.03 under a predeclared 0.05 threshold and explain what it does not tell you. Finally describe why measuring twenty outcomes and promoting only the smallest unadjusted p-value changes the error-control claim.

Worked solution

One defensible design assigns accounts once, measures whether each assigned user completes a specified lesson within seven days, and analyzes after a predeclared sample or enrollment period plus full follow-up. Count noncompletion according to that binary outcome rather than dropping users who leave; separately document missing telemetry. Under the test's assumptions, p=0.03 is below the chosen threshold and supports rejecting the specified null. It does not mean a 97% probability that the new layout is better, nor does it establish a useful effect size or freedom from bias. Searching twenty outcomes raises the opportunity for a small p-value under null effects. Use a predeclared primary outcome or an appropriate multiple-testing procedure, and report the whole comparison set rather than selectively presenting the most favorable result.

Check your understanding

What is a p-value conditioned on?

• The null model and test assumptions being used.
• The alternative definitely being true.
• The observed result having no measurement error.

Explanation

It calculates the probability of sufficiently extreme data under the specified null model, not the posterior probability of that model.

Why can stopping at the first p-value below 0.05 invalidate an ordinary fixed-sample test guarantee?

• It changes the sampling and decision procedure through repeated opportunities to reject.
• More data always makes tests invalid.
• A p-value can never be recomputed.

Explanation

Sequential monitoring requires a procedure whose error control accounts for the stopping rule.

Primary reading

• Introduction to Probability and Statistics: MIT OpenCourseWare

Open the study and save your work →