Classification and decisions

Prerequisites: The probability and linear-model chapters; logarithms are introduced through examples.

• Calculate sigmoid probabilities and binary log loss.
• Build a confusion matrix at two thresholds.
• Explain why calibration, ranking, and decisions differ.

A score, a probability, and an action

A binary classifier predicts whether y is 0 or 1. Logistic regression first forms z=w·x+b, then computes p=1/(1+exp(−z)). This sigmoid maps any finite score into (0,1). At z=0, p=0.5; at z=ln(3), p=0.75. Interpret p as a model’s probability estimate, whose calibration still needs evaluation. A threshold turns that estimate into a decision.

Binary log loss for one example is −[y ln(p)+(1−y) ln(1−p)]. It penalizes confident incorrect predictions strongly. For y=1, p=0.8 gives loss about 0.223; p=0.2 gives about 1.609. The logarithm here is natural. Google’s logistic-regression lesson explains this fitting objective and regularization.

Derive the update rather than memorizing it

The sigmoid derivative is p(1−p). The loss derivative with respect to p is −y/p+(1−y)/(1−p). Multiplying and simplifying gives dL/dz=p−y. Therefore dL/dw_j=(p−y)x_j and dL/db=p−y for a single example. For a batch mean, average these derivatives across examples.

Take x=[2,−1], w=[0,0], b=0, and y=1. Then z=0, p=0.5, dL/dz=−0.5, and the weight gradient is [−1,0.5]. A simultaneous step of size 0.1 yields w=[0.1,−0.05], b=0.05. The new score is 0.3 and the new probability is about 0.5744. The model moves toward the observed label; one example does not establish general usefulness.

For numerical stability, calculate single-example loss from the score as max(z,0)−y*z+ln(1+exp(−abs(z))). This equals the same binary log loss while avoiding exp(z) for large positive z. It also avoids taking the logarithm of a probability rounded to exactly zero.

Count the errors a threshold produces

At a chosen threshold, true positives are actual positives predicted positive; false positives are actual negatives predicted positive. False negatives are missed positives, and true negatives are correctly rejected negatives.

precision = TP / (TP + FP)
recall = TP / (TP + FN)
false-positive rate = FP / (FP + TN)
accuracy = (TP + TN) / (TP + FP + FN + TN)

Each denominator asks a different question. Precision concerns the set of positive predictions; recall concerns the set of actual positives. Report undefined ratios when a denominator is zero, or state an explicit software convention. Google’s classification metrics reference defines these quantities and their dependence on the decision threshold.

Suppose a dataset has 100 examples, of which 10 are positive. A classifier yields TP=7, FN=3, FP=5, TN=85. Accuracy is 92%, precision is 7/12≈58.33%, and recall is 70%. An always-negative baseline reaches 90% accuracy but detects no positives. The two-point accuracy improvement conceals a large change in which errors occur.

Ranking quality is not calibration

A model ranks well when useful examples tend to receive higher scores. Calibration asks whether a group assigned probability near p actually has a positive frequency near p, within sampling uncertainty. Replacing every score by its square preserves ranking on [0,1] but usually changes calibration. At a fixed numeric threshold, it also changes decisions.

If a calibrated probability p is available and the only errors cost C_FP and C_FN, predicting positive has expected cost C_FP(1−p), while predicting negative costs C_FN*p. The cost-optimal threshold is C_FP/(C_FP+C_FN), with a tie at equality. Capacity limits, uncertain costs, or poorly calibrated scores require additional analysis. On a finite dataset, you can compare candidate thresholds using validation costs directly.

Complete the exercise with pencil and paper. Write which examples change classification as the threshold falls. A table of changed cases is a better debugging aid than one aggregate percentage.

Practice exercise

Use scores [0.9, 0.7, 0.6, 0.4, 0.2, 0.1] and labels [1,0,1,1,0,0]. Predict positive when score≥threshold. Calculate TP, FP, FN, TN, precision, recall, and cost at thresholds 0.5 and 0.3. Use false-positive cost 1 and false-negative cost 4. Choose between these thresholds using this validation set.

Compare with the worked solution

At 0.5: TP=2, FP=1, FN=1, TN=2; precision=2/3, recall=2/3, cost=1+4=5. At 0.3: TP=3, FP=1, FN=0, TN=2; precision=3/4, recall=1, cost=1. On these validation examples, choose 0.3. Both thresholds have false-positive rate 1/3. Six observations are too few to establish stable deployment performance. Lock the chosen threshold before evaluating on a separate test set.

Check your understanding

What is recall at threshold 0.3 in this exercise?

• 1/2
• 2/3
• 1
• 3/4

Check the reasoning

All three actual positives are detected: recall=TP/(TP+FN)=3/3.

A model assigns probability 0.99 to a positive label that is actually 0. What does binary log loss do?

• It ignores the mistake.
• It is zero because the confidence is high.
• It is negative.
• It assigns a large penalty, about 4.605.

Check the reasoning

For label 0, loss is −ln(1−p). At p=0.99, that is −ln(0.01)≈4.605.

Learn AI curriculum · Next: Generalization and leakage