Random variables, expectation, and variance

Prerequisites: Probability, finite sums, and algebra.

Learning objectives

• Compute expectation and variance from a distribution.
• Use linearity of expectation without assuming independence.
• Identify when covariance changes the variance of a total.

A random variable assigns a number to an outcome

A random variable is a function on the sample space. It might record request latency, the number of cache misses, or a simulation's score. For a discrete variable X, its distribution specifies P(X=x), and its expectation is the weighted sum of values: E[X]=sum of x*P(X=x), when that expectation exists.

Suppose a request takes 1 millisecond with probability 0.8 and 6 milliseconds with probability 0.2. Its mean latency is 10.8+60.2=2 milliseconds. No request in this model actually takes 2 milliseconds. An expected value is a distribution summary, not necessarily a possible outcome and not a bound on what one observation will do.

Variance measures expected squared deviation from the mean. Here Var(X)=0.8*(1-2)^2+0.2*(6-2)^2=4 square milliseconds, giving standard deviation 2 milliseconds. The alternative expression E[X^2]-(E[X])^2 yields the same mathematical value: 0.81+0.236-4=4. Numerical implementations must still consider rounding when subtracting nearby large quantities.

Expectations add even under dependence

For random variables with suitable finite expectations, E[X+Y]=E[X]+E[Y]. Independence is not required. If a batch contains 100 requests each with mean latency 2 milliseconds, the expected sum of their individual latencies is 200 milliseconds. That is not automatically the batch's wall-clock duration when requests overlap or queue. Define the quantity being measured before applying the formula.

Indicator variables make counting simple. Let I_i equal one when item i has a property and zero otherwise. Then E[I_i]=P(item i has the property), and the expected count is the sum of those probabilities. Dependence between indicators does not invalidate this expectation calculation.

Variances need dependence information

Var(X+Y)=Var(X)+Var(Y)+2Cov(X,Y). Independent variables have zero covariance when these moments exist, so their variances add. Correlated delays from a shared outage can make totals much more variable than an independence calculation suggests. Zero covariance alone does not generally imply independence.

For n independent, identically distributed observations with finite variance sigma squared, the sample mean has variance sigma squared divided by n. Its standard deviation shrinks like one over the square root of n. Four times as many independent observations halves that standard deviation, rather than dividing it by four. This relationship underlies uncertainty estimates, but repeated measurements of the same correlated event do not create the same information as independent observations.

Practice problem

A fair coin determines two indicators X and Y. In the first experiment, both equal one on heads and zero on tails. In the second, X indicates heads and Y indicates tails. For each experiment compute E[X+Y] and Var(X+Y), and compare with Var(X)+Var(Y). Explain why expectation additivity survives both dependence patterns while variance additivity fails. Then state the variance of the average of 16 independent copies of a variable whose variance is 9.

Worked solution

Each indicator has mean 1/2 and variance 1/4. In the first experiment the sum is either zero or two with equal probability, so its mean is one and its variance is one. The sum of individual variances is only 1/2; positive covariance supplies the remaining 1/2. In the second experiment the sum is always one, so its mean remains one but its variance is zero. Negative covariance cancels the individual variances. Linearity of expectation requires no independence, whereas removing covariance from a variance calculation does. For 16 independent copies with variance 9, the mean has variance 9/16 and standard deviation 3/4. If the copies were perfectly correlated, averaging them would not provide this reduction.

Check your understanding

Which statement requires no independence assumption, assuming expectations exist?

• E[X+Y]=E[X]+E[Y].
• Var(X+Y)=Var(X)+Var(Y).
• Observing X never changes the distribution of Y.

Explanation

Linearity of expectation holds under dependence; the other statements need additional conditions.

For independent identically distributed observations with finite variance, quadrupling sample size changes the mean standard deviation by what factor?

• One quarter.
• One half.
• Four.

Explanation

The standard deviation of the mean scales as 1/sqrt(n).

Primary reading

• Statistics 110: Probability: Harvard University
• Introduction to Probability and Statistics: MIT OpenCourseWare

Open the study and save your work →