FREE COMPUTER SCIENCE STACK LEVEL · 3 OF 20

Probability & statistics

What can observations justify, and how uncertain is the conclusion?

The question this layer answers

What can observations justify, and how uncertain is the conclusion?

Reason under uncertainty. This layer connects Bayes, Random variables, Estimation, Experiments, Regression to the rest of the computing stack.

Reason under uncertainty

Work with distributions, conditional information, sampling, and uncertainty. Learn to distinguish a measured association from a causal claim and numerical precision from reliable inference.

What can observations justify, and how uncertain is the conclusion?

This section is a sequence of worked studies. Each study asks you to reconstruct an example, solve a new problem, and check the assumptions behind your answer. Keep your derivations and code so you can revisit them without looking at the solution.

What you will work through

1. Conditional probability and Bayes reasoning

Update beliefs using base rates, likelihoods, and an explicit sample space. Compute conditional probabilities with the correct denominator. Apply total probability and Bayes rule. Distinguish independence from mutually exclusive events.

2. Random variables, expectation, and variance

Measure outcomes and reason about totals without confusing averages with guarantees. Compute expectation and variance from a distribution. Use linearity of expectation without assuming independence. Identify when covariance changes the variance of a total.

3. Sampling, estimation, and confidence intervals

Separate population targets, sampling bias, and uncertainty caused by finite data. Distinguish a population parameter from an estimator. Interpret a frequentist confidence procedure correctly. Identify uncertainty that a standard error does not measure.

4. Experiments, hypothesis tests, and effect sizes

Design comparisons that answer a causal question and report uncertainty honestly. Explain the role of random assignment. Interpret a p-value without reversing its condition. Separate statistical evidence, effect magnitude, and practical importance.

5. Regression, prediction, and confounding

Fit relationships while separating predictive association from causal effect. Compute and interpret a simple least-squares fit. Diagnose residual patterns and extrapolation risk. Explain confounding and why predictive accuracy is not causal identification.

6. Floating-point stability and problem conditioning

Separate rounding introduced by an algorithm from sensitivity inherent in the problem. Distinguish absolute and relative error. Explain catastrophic cancellation and a stable reformulation. Separate a poorly conditioned problem from an unstable algorithm.

Before you begin

Fractions, sets, counting, and elementary algebra. Each lesson names any additional prerequisites. Revisit the mathematics and programming sections whenever a calculation or execution trace becomes a barrier.

Evidence of understanding

Analyze a reproducible dataset or simulation: state the sampling process, compute estimates and uncertainty, test assumptions, and explain one conclusion the evidence cannot support. A correct multiple-choice answer is only one kind of evidence. Preserve the intermediate steps, test an edge case, and explain why an attractive alternative fails.

Continue beyond the guided lessons

The primary readings provide fuller treatments and additional exercises. Work the section project after the guided problems, then return to a problem a few days later without notes. Use a failed attempt to identify the specific definition, assumption, or step that needs more practice.

Free Probability & statistics lessons

Practice lab: Update a belief with a noisy signal

About 35 minutes. Fractions and conditional probability.

One percent of components are defective. A test flags 90% of defective components and 5% of nondefective components. Model 10,000 independent components with those exact expected proportions.

  1. Build a two-by-two expected-count table.
  2. Calculate the probability of defect given a flag.
  3. Repeat with defect prevalence 10%, keeping the test behavior fixed.
Hint

Condition on all flagged components, including false positives.

Worked solution

At 1% prevalence, expect 100 defective with 90 flagged, and 9,900 nondefective with 495 flagged. The conditional probability is 90/585, about 15.38%. At 10%, expect 900 true flags and 450 false flags, so it is 900/1350=66.67%. The same sensitivity and false-positive rate produce a different posterior when the prior prevalence changes. Expected counts need not equal an observed random sample.

  • Include the nondefective population in the denominator.
  • Distinguish sensitivity from posterior probability.
  • State the prevalence assumption.

Change one assumption

What additional assumption is needed before treating two repeated test results as conditionally independent?

Reason with uncertainty and imperfect measurements

Calculate from explicit sample spaces, separate sampling error from bias, and test whether a numerical method preserves the information in its input. State which assumptions make each conclusion possible.

1. Conditional probability · allow 35 minutes

A detector that changes meaning across populations

In a population of 10,000 items, exactly 100 are defective. A detector flags 95 defective items and 5% of the good items. Build the full count table and compute P(defect|flag), P(defect|no flag), sensitivity, and specificity.

Repeat the positive predictive value calculation for a population with 20% defective items, assuming the same sensitivity and specificity. Explain what has changed. If replacing a good item costs 4 and accepting a defective item costs 60, derive a decision threshold from an estimated defect probability p. State why the detector's binary flag alone may be insufficient for other cost structures.

Get a starting hint

Conditioning changes the denominator. Derive the action threshold by comparing expected losses, not by choosing a conventional probability cutoff.

Compare your reasoning with the worked solution

The first population has TP 95,FN 5,FP 495,TN 9405. P(defect|flag)=95/590≈16.10%; P(defect|no flag)=5/9410≈0.0531%. Sensitivity is 95/100=95%, specificity 9405/9900=95%. With 20% prevalence in 10,000 items, TP 1900,FN 100,FP 400,TN 7600, so positive predictive value is 1900/2300≈82.61%. Sensitivity and specificity were held fixed; the prior probability changed.

Replacing has expected loss 4(1−p), while accepting has 60p. Replace when 4(1−p)<60p, giving p>1/16=6.25%, with indifference at equality. Under the first population's assumptions a positive flag exceeds this threshold, while a negative flag does not. Different costs can change that policy even with identical detector accuracy. A binary output cannot rank risks within its positive and negative groups. Transferring the detector to another population also requires checking, rather than assuming, that its conditional error rates remain stable.

Check your reasoning

  • Make all four cells sum to the stated population.
  • Use the flagged denominator for precision.
  • State both the prevalence assumption and the cost model.

2. Expectation and dependence · allow 35 minutes

When redundant components fail together

A service uses two components. Each is unavailable with probability 0.02 in a randomly selected observation interval. The service is unavailable only if both are unavailable. Compute service unavailability if the events are independent.

Now suppose a common power event occurs with probability 0.01 and disables both. Conditional on no power event, components fail independently with probability q. Choose q so each component still has marginal unavailability 0.02. Recalculate service unavailability. For two Bernoulli failure indicators X,Y, compare E[X+Y] and Var(X+Y) between the two models.

Get a starting hint

Marginal failure includes common-power intervals. Use the law of total probability before multiplying conditional probabilities.

Compare your reasoning with the worked solution

With independence, joint unavailability is 0.02²=0.0004. In the common-cause model,0.01+0.99q=0.02, so q=1/99≈0.010101. Joint unavailability is 0.01+0.99q²=1/99≈0.010101, about 25.25 times the independent result.

In both models E[X+Y]=E[X]+E[Y]=0.04; linearity of expectation requires no independence. Each marginal variance is 0.02×0.98=0.0196. Independence gives total variance 0.0392. With shared power, Cov(X,Y)=E[XY]−E[X]E[Y]=1/99−0.0004≈0.009701, so variance is 0.0392+2×0.009701≈0.058602.

The components' identical individual reliability does not establish the joint behavior needed for service availability. Independence is a structural assumption about failure causes. Adding replicas inside one failure domain may add much less protection than a multiplication of marginal probabilities suggests.

Check your reasoning

  • Solve q from the marginal requirement.
  • Separate unconditional and conditional independence.
  • Explain why expectation is unchanged while variance and joint failure change.

3. Estimation and bias · allow 40 minutes

A narrow interval around the wrong population

A random sample of 100 completed requests has mean latency 12 ms and sample standard deviation 4 ms. Using a normal approximation with multiplier 1.96, estimate the mean's standard error and a 95% confidence interval. State the sampling assumptions and interpret the procedure correctly.

You then learn that 20 timed-out requests were omitted. For this exercise their latencies are only known to lie between 0 and 40 ms. Bound the mean over all 120 requests. Explain why collecting more completed requests may narrow the original interval without repairing this selection problem.

Get a starting hint

The standard error uses the square root of sample size. A bound for missing observations is a different calculation from a confidence interval.

Compare your reasoning with the worked solution

The estimated standard error is 4/√100=0.4 ms. The approximate interval is 12±1.96×0.4, or[11.216,12.784] ms. It assumes a sampling scheme and observations for which this approximation is appropriate, including sufficient independence and a suitable distribution of the sample mean. Across repeated samples, the method aims for 95% coverage; after observing this interval the fixed population mean is not assigned a frequentist 95% probability by this calculation alone.

The observed total is 1200 ms. Missing latencies contribute between 0 and 800 ms, so the combined mean is bounded by[1200/120,2000/120]=[10,16.6667] ms. These are identification bounds under the supplied range, not 95% confidence limits. In a real timeout system the timeout threshold could impose a stronger lower bound, but it was not given here.

A larger sample from the same selected population may estimate the mean conditional on completion more precisely. It does not make that conditional mean equal the mean of all requests. Sampling uncertainty, measurement error, and selection bias require separate arguments.

Check your reasoning

  • Label the interval as approximate and state its sampling assumptions.
  • Compute bounds from totals, including the missing observations.
  • Do not describe a confidence interval as a cure for selection bias.

4. Experimental design · allow 45 minutes

Reconcile subgroup and pooled results

A team assigns a new interface mainly to experienced users. Results are: new interface, experienced 90 successes/100 users and beginners 4/20; old interface, experienced 19/20 and beginners 50/100. Compute subgroup rates and pooled rates. Standardize both interfaces to a population that is half experienced and half beginner.

Explain why the pooled comparison does not identify the interface's causal effect. Design a randomized experiment that avoids this allocation problem. Separately, an actual randomized trial records 520/1000 successes with new and 500/1000 with old. Calculate the observed difference and an approximate 95% interval using independent Bernoulli standard errors.

Get a starting hint

Changing subgroup weights changes the pooled rate. For the randomized trial use SE=sqrt(p1(1−p1)/n1+p0(1−p0)/n0).

Compare your reasoning with the worked solution

The new interface rates are 90% and 20%; the old rates are 95% and 50%. Pooled rates are 94/120≈78.33% versus 69/120=57.5%, apparently favoring new despite worse performance in both subgroups. Equal-weight standardized rates are 0.5×0.90+0.5×0.20=55% and 0.5×0.95+0.5×0.50=72.5%. Assignment and user experience are entangled, so the pooled association cannot isolate the interface effect. Even adjustment requires assumptions about other differences.

Randomize within experience strata, record allocation before outcomes, use the same task and observation window, and predeclare the outcome and stopping rule. For the separate randomized data the difference is 0.52−0.50=0.02. SE=√(0.2496/1000+0.25/1000)≈0.02235, giving approximately[−0.02381,0.06381]. The data are compatible with both a small decrease and a modest increase at this precision. The interval is not a probability that the effect lies inside, and failure to exclude zero does not prove equivalence.

Check your reasoning

  • Calculate rates within groups before pooling.
  • Use the specified common weights for standardization.
  • Separate a causal identification problem from statistical imprecision.

5. Model critique · allow 35 minutes

A perfect prediction with the wrong explanation

Across three regions, recorded advertising spend x is 1,2,3 million and revenue y is 10,20,30 million. Fit y=a+bx by least squares and give residuals. A manager predicts that adding 1 million of spend to the first region will increase revenue by 10 million.

Construct an alternative data-generating explanation using region population that fits all three observations but has no causal effect of advertising. State what a randomized geographic experiment would need to compare. Explain how training and test sets could both look excellent while the causal claim remains unsupported.

Get a starting hint

Let an unobserved common cause determine both variables. Predicting one variable from another does not identify the effect of intervening on it.

Compare your reasoning with the worked solution

The fitted line is a=0,b=10, with all residuals zero. One compatible model has region population z=1,2,3 million, advertising budget x=z, and revenue y=10z. In that model changing x while holding z fixed does not change y. Another model could include an advertising effect. The observations alone cannot distinguish those causal structures.

An experiment would randomly assign an advertising change across comparable geographic units, define the revenue window, measure spillovers and baseline differences, and estimate effects at the assigned intervention level. Too few independent regions can make uncertainty large even with many customers per region.

A held-out dataset generated by the same population-to-budget policy can preserve the relationship y≈10x and yield excellent predictive accuracy. Such a split tests prediction under that distribution, not the response to a changed budgeting policy. Extrapolating beyond observed budgets introduces an additional assumption. Report the fitted association and the intervention question separately.

Check your reasoning

  • Calculate the line and residuals explicitly.
  • Give a complete alternative mechanism consistent with the data.
  • Identify the unit of randomization and distinguish prediction from intervention.

6. Numerical analysis · allow 45 minutes

Fix an unstable formula without changing the problem

For x>0, evaluate g(x)=sqrt(x²+1)−x. Derive an algebraically equivalent expression that avoids subtracting two nearly equal large numbers. Evaluate both in binary64 arithmetic at x=10^8 and compare with the approximation 1/(2x).

Explain why rearrangement helps and name a separate overflow risk for extremely large x. Then compare computing (10^16+1)−10^16 with 10^16+(1−10^16). Distinguish representation error, operation ordering, algorithmic stability, and problem conditioning.

Get a starting hint

Multiply g by the conjugate. The square-root operation and intermediate x² still require range analysis.

Compare your reasoning with the worked solution

Multiplying numerator and denominator by sqrt(x²+1)+x gives g(x)=1/(sqrt(x²+1)+x). At x=10^8, binary64 may round x²+1 to x², making the direct subtraction zero. The rearranged expression gives about 5×10^−9, matching the leading approximation 1/(2x). Algebraic equality over real numbers does not imply identical floating-point evaluation.

For much larger x, forming x² can overflow even when the final answer is representable. A range-safe hypotenuse computation, such as a suitably implemented hypot(x,1), addresses that intermediate risk; the sum in the denominator also needs consideration near the largest representable values.

Under usual binary64 rounding, (10^16+1)−10^16 is 0 because the addition loses 1. The second expression is also 0 in this particular ordering, so it is not a useful associativity counterexample. Compare instead (10^16−10^16)+1=1 with 10^16+(−10^16+1)=0. A careful numerical critique checks the proposed counterexample rather than repeating a slogan. Instability describes unnecessary error introduced by an algorithm; conditioning describes sensitivity of the mathematical answer to small input perturbations.

Check your reasoning

  • Derive the conjugate form and test both implementations.
  • Separate cancellation from overflow.
  • Verify the actual floating-point example before claiming non-associativity.

Study the underlying methods

Section project: Run a reproducible experiment and uncertainty audit

5 to 7 hours. Build a simulation study of a randomized learning intervention with binary completion outcomes. Inputs are a seed, users per group, baseline completion probability, treatment effect, and number of repeated experiments. Randomly assign synthetic users to control or treatment, draw outcomes from the declared Bernoulli model, and report the difference in completion proportions with an interval procedure whose assumptions you state. Use a randomization test that permutes treatment labels while preserving group sizes; compute a two-sided p-value with a finite-permutation correction so it is never spuriously zero. Before inspecting results, write a decision rule, primary outcome, and fixed sample-size plan. Run repeated null experiments to measure the empirical rejection rate, then repeat with a nonzero effect to study power. Finally introduce a deliberately biased analysis that discards noncompleters or repeatedly peeks at ordinary fixed-sample tests, and explain the changed procedure. Python’s random, math, statistics, and csv modules are enough. Submit reproducible outputs and a short report separating assumptions, effect size, uncertainty, and causal interpretation within this synthetic model. Choose an interval method that states its behavior at zero and complete success counts. Keep interval coverage claims separate from the randomization test, since the two procedures can rely on different assumptions.

Milestones

  1. Specify the data-generating model, assignment scheme, estimand, primary test, and stopping rule.
  2. Implement outcome generation, effect estimation, and a documented confidence procedure.
  3. Implement label permutation with a two-sided statistic and finite-simulation p-value correction.
  4. Run repeated null and alternative experiments and report Monte Carlo uncertainty for rejection rates.
  5. Demonstrate one biased analysis and explain which original guarantee no longer applies.

Acceptance criteria

  • Repeating the same seed and configuration reproduces assignments, outcomes, and analysis outputs.
  • Every permutation preserves the original group sizes and reassigns only labels, not outcome values.
  • The reported Monte Carlo p-value uses (extreme_count+1)/(permutation_count+1) and lies strictly above zero.
  • The effect estimate equals treatment completions divided by treatment size minus the corresponding control proportion.
  • Null rejection-rate results include the number of repetitions and a stated measure of Monte Carlo uncertainty rather than demanding exact equality with alpha.
  • The report identifies the biased analysis as a changed selection or stopping procedure and does not interpret a p-value as the probability that the null is true.

Introduce correlated outcomes within study groups and compare individual-level randomization analysis with a cluster-aware design, explaining the effective sampling unit.

Practise Probability & statistics free →