Data, model, evaluation

The historical question

Modern capability joined large datasets, labeling labor, accelerator compute, model architecture, optimization, evaluation, and post-training. Failures exposed that aggregate benchmark performance can hide uneven behavior and unexamined data choices.

Learned behavior is produced by the complete data-objective-model-evaluation system, so no model result can be interpreted apart from its evidence pipeline.

An evaluation is a measurement design

Consider a classifier evaluated on 100 examples,90 negative and 10 positive. Always predicting negative achieves 90% accuracy but detects none of the positive cases. If missing a positive is costly, that headline accuracy does not answer the operational question. Report the confusion matrix and the costs or goals that make particular errors matter.

Now suppose examples from the same person occur in both training and testing. A model may exploit person-specific details that will not help on a new person. A random split can then overestimate the intended generalization. Choose the split around the deployment question: new people, future time periods, new environments, or some combination. Keep the final test independent of repeated model and threshold selection.

Historical progress in learned systems combines datasets, labeling work, hardware, architectures, optimization, and evaluation. A benchmark improvement is evidence for a result under the reported protocol; it is not automatically evidence of universal competence or equal performance for every group. Compare disaggregated error rates, sample sizes, and uncertainty. A small subgroup can have a highly unstable estimate. Claims of fairness or human-like reasoning require additional definitions and evidence beyond a single aggregate score.

What the evidence does not establish

This is provisional history involving living people and contested institutions. Teach documented mechanisms and lineages, add review dates, and avoid settled heroic verdicts.

Reconstruction exercise

For 90 true negatives and 10 false negatives, compute accuracy and recall of the positive class. Explain why a person-disjoint test might be needed.

Worked solution

Accuracy is 90/100 = 90%. Positive recall is 0/(0+10) = 0%. A person-disjoint split tests transfer to individuals absent from training rather than recognition of familiar individuals. The appropriate split depends on intended use, and even a well-designed split does not eliminate distribution shift. Report both the metric and the population it measures.

Test the limits and transfer the idea

1. Confusion matrix and decision analysis · allow 30 minutes

Higher accuracy can have higher stated error cost

A fictional test has 950 negatives and 50 positives. Model A has TP=40 and FP=95; model B has TP=25 and FP=19. Compute FN, TN, accuracy, positive precision and recall for both. Under the stipulated cost of 20 units per false negative and one per false positive, compare total cost. Explain why the choice may change if deployment prevalence or costs differ.

Get a starting hint

Fill each confusion matrix before comparing metrics; do not use accuracy as a substitute for the declared objective.

Compare your reasoning with the worked solution

A has FN=10,TN=855, accuracy 895/1000=89.5%, precision 40/135≈29.63% and recall 40/50=80%. B has FN=25,TN=931, accuracy 956/1000=95.6%, precision 25/44≈56.82% and recall 25/50=50%. Costs are 20×10+95=295 for A and 20×25+19=519 for B, so A is preferable under this specific additive cost model despite lower accuracy. Deployment prevalence changes the mixture of positive and negative cases and thus expected cost and precision. Costs may also differ across consequences. The cited research papers are primary reports of their own datasets and evaluations; these invented matrices establish no numerical claim about ImageNet, Gender Shades or deployed products.

Check your reasoning

• Completes both matrices and calculates all requested metrics.
• Computes costs 295 and 519 under the given model.
• States the prevalence and objective limits of the comparison.

2. Sampling calculation and source criticism · allow 30 minutes

Zero observed errors do not prove a zero error rate

A fictional subgroup test has zero errors among 40 independent representative cases. If the true error probability were 5%, calculate the probability of observing zero errors. Find p such that (1−p)^40=0.05 and interpret that boundary cautiously. Explain two reasons the independent-representative assumption could fail. Relate the question to why a primary evaluation paper's dataset and subgroup definitions must be inspected rather than only its headline.

Get a starting hint

A finite sample can contain no errors even when the underlying probability is positive.

Compare your reasoning with the worked solution

At true p=0.05, zero-error probability is 0.95⁴⁰≈0.1285, about 12.85%, so the observation is not especially surprising. Solving gives p=1−0.05^(1/40)≈0.07216. Under the stated binomial sampling model, this is the usual one-sided 95% upper confidence boundary for zero observed errors; it is not a 95% posterior probability that the unknown rate lies below it. Repeated images of the same person can violate independence, and a convenient collection can omit difficult deployment conditions, violating representativeness. Gender Shades is a primary study with an explicit evaluation design, not evidence that this invented forty-case sample or every future classifier has the same properties.

Check your reasoning

• Computes about 12.85% and the 7.216% boundary.
• Avoids interpreting a confidence boundary as a posterior probability.
• Identifies dependence and selection failures in the sample design.

Check your understanding

What does 90% accuracy establish in this example?

• That 90 of 100 labels match, despite zero positive recall
• That 90% of positives are detected
• That error costs are acceptable

Explanation

Accuracy is 90/100 =90%. Positive recall is 0/(0+10) =0%. A person-disjoint split tests transfer to individuals absent from training rather than recognition of familiar individuals. The appropriate split depends on intended use, and even a well-designed split does not eliminate distribution shift. Report both the metric and the population it measures.

Sources and further reading

• ImageNet: IEEE CVPR · primary paper
• Attention Is All You Need: NeurIPS
• Gender Shades: Proceedings of Machine Learning Research · original paper
• Training language models to follow instructions with human feedback: Ouyang et al. · primary paper