Usability evidence and controlled experiments

Prerequisites: Sampling, estimation, and experimental design.

Learning objectives

• Distinguish usability tests from A/B tests.
• Define a metric and guardrail.
• Recognize selection and novelty effects.

A metric operationalizes a question

“Make the course better” is too broad to test directly. A more precise question is whether a new navigation design helps returning learners find the next appropriate exercise. Possible measures include successful task completion, time to locate the exercise, incorrect starts, and the learner's understanding of what remains unfinished.

A usability study examines how and why interaction succeeds or fails. A randomized experiment estimates the effect of assigning a variant on a specified outcome under its design assumptions. These methods complement one another. An A/B test may detect a difference without explaining its cause; observation may reveal a cause without estimating its prevalence.

Work an experiment definition

Suppose eligible returning users are randomly assigned to the existing dashboard or a revised one. The primary metric is the fraction who resume an unfinished exercise within ten minutes of arrival. Define eligibility, assignment unit, observation window, and what counts as resuming before looking at results. Count all assigned eligible users according to the intended analysis rather than excluding inconvenient failures after the fact.

Add guardrails such as accidental restarts, error rates, and performance. More clicks or more time on site is not necessarily better learning. A confusing page can increase both. If the intended outcome is retention of knowledge, include a later assessment or another justified learning measure.

Interpret differences with their design

Random assignment helps separate the treatment effect from preexisting differences in expectation. It does not fix missing outcomes, cross-group interference, broken instrumentation, or a sample that does not represent the intended deployment. Users sharing devices can complicate account-level assignment. Repeatedly checking results and stopping at the first favorable threshold can invalidate a naive fixed-sample analysis.

Report effect size and uncertainty, not only a threshold decision. A small statistically detectable effect can be practically irrelevant; a large estimate from a tiny sample can be too uncertain to trust. Segment analysis should follow a rationale and avoid treating every exploratory fluctuation as a discovery.

Learning is more than immediate conversion

A design that makes answer submission effortless may improve completion while reducing thought. Compare immediate engagement with delayed recall, transfer to unfamiliar problems, and error recovery where feasible. State the trade-off openly. The purpose of measurement is to support a decision about the learner's task, not to manufacture a favorable number for an already chosen interface.

Practice problem

Design an experiment comparing “show solution immediately” with “attempt first, then reveal.” Choose a primary learning outcome, one engagement guardrail, and one reason immediate lesson completion could mislead.

Worked solution

Randomly assign eligible learners to the two flows and use performance on a later equivalent but unseen problem as the primary outcome. Define timing, scoring, and handling of missing follow-up responses in advance. Track abandonment or inability to proceed as a guardrail, and ensure either flow remains accessible. Immediate completion can rise because learners reveal and copy answers without retrieving or applying the concept. The experiment should report uncertainty, follow-up participation, and whether the result applies beyond the tested topic and learners.

Check your understanding

Why can time on page be a poor success metric?

• Time cannot be measured.
• Longer time may reflect confusion rather than learning.
• All learning is instantaneous.
• It always decreases with better design.

Explanation

The same metric can arise from productive study or avoidable friction.

What does random assignment primarily help address?

• Every measurement bug.
• All missing data.
• Preexisting differences between treatment groups in expectation.
• Whether the metric captures learning.

Explanation

Randomization supports causal comparison but does not define a meaningful outcome.

Primary reading

• Introduction to Human-Computer Interaction: University of Washington CSE 440
• Web accessibility tutorials: W3C Web Accessibility Initiative

Open the study and save your work →