A reproducible experiment

Prerequisites: Chapters 2 through 5 and a local Python 3 interpreter. No packages, account, API, or GPU required.

• Run a complete training, validation, and test workflow.
• Compare with a constant baseline using costs and counts.
• Document what synthetic evidence cannot establish.

The experiment’s question

Build a logistic classifier for a synthetic binary label from two numeric features. The purpose is to reconstruct the learning procedure: generate examples, partition them, fit parameters, select a threshold, and evaluate once. The synthetic label rule is known, so this exercise cannot establish usefulness on a real population. It can expose implementation and evaluation mistakes.

Use Python 3 locally. Save the program below as experiment.py and run python3 experiment.py. It uses only the standard library. There are no downloads, credentials, external services, or charges. The page does not execute code or grade your local output.

Fix the rules before running

The generator draws two features independently and uniformly from [−1,1], calculates a probability with a logistic rule, and draws a binary label from it. A local seeded random generator makes the data reproducible within this program. We use 300 examples for training, 150 for validation, and 150 for test. Because the generator produces independent examples from one fixed rule, this simple partition matches the toy generalization question.

We minimize mean binary log loss with full-batch gradient descent. No feature scaler is fitted because the generated features already share the same bounded scale. We compare three threshold candidates on validation cost, charging one unit for a false positive and four for a false negative. The lowest validation cost wins; ties prefer the larger threshold. The test set is evaluated only after that choice.

A complete implementation

from math import exp, log1p
from random import Random

def sigmoid(z):
    if z >= 0:
        return 1 / (1 + exp(-z))
    e = exp(z)
    return e / (1 + e)

def log_loss(z, y):
    return max(z, 0) - y*z + log1p(exp(-abs(z)))

def dataset(size, seed):
    rng = Random(seed)
    rows = []
    for _ in range(size):
        x = [rng.uniform(-1, 1), rng.uniform(-1, 1)]
        probability = sigmoid(2*x[0] - 1.5*x[1] - 0.4)
        y = int(rng.random() < probability)
        rows.append((x, y))
    return rows

def score(x, w, b):
    return sum(a*v for a, v in zip(w, x)) + b

def fit(rows, steps=1200, rate=0.1):
    w, b = [0.0, 0.0], 0.0
    for _ in range(steps):
        dw, db = [0.0, 0.0], 0.0
        for x, y in rows:
            error = sigmoid(score(x, w, b)) - y
            for j in range(2):
                dw[j] += error*x[j] / len(rows)
            db += error / len(rows)
        w = [v-rate*g for v, g in zip(w, dw)]
        b -= rate*db
    return w, b

def counts(rows, w, b, threshold):
    result = dict(tp=0, fp=0, fn=0, tn=0)
    for x, y in rows:
        predicted = sigmoid(score(x, w, b)) >= threshold
        key = ('tp' if y else 'fp') if predicted else ('fn' if y else 'tn')
        result[key] += 1
    return result

def cost(result, fn_cost=4):
    return result['fp'] + fn_cost*result['fn']

rows = dataset(600, seed=17)
train, validation, test = rows[:300], rows[300:450], rows[450:]
w, b = fit(train)
training_loss = sum(log_loss(score(x, w, b), y) for x, y in train)/len(train)
assert training_loss < log_loss(0, 0)  # Better than initial p=0.5 on training.

candidates = [0.2, 0.35, 0.5]
chosen = min(candidates, key=lambda t: (cost(counts(validation, w, b, t)), -t))
print('Chosen threshold:', chosen)
print('Training loss:', round(training_loss, 4))
test_counts = counts(test, w, b, chosen)
assert sum(test_counts.values()) == len(test)
print('Test counts:', test_counts)
print('Test cost:', cost(test_counts))

positives = sum(y for _, y in test)
baseline = dict(tp=0, fp=0, fn=positives, tn=len(test)-positives)
print('Always-negative baseline:', baseline)
print('Baseline cost:', cost(baseline))

Explain the output without overselling it

Keep the selected threshold, test counts, and baseline counts together. Derive precision and recall from the counts, stating what you do when a denominator is zero. Raw error cost depends on sample size, so include the number of test examples; divide by that count if reporting mean cost.

The theoretical threshold for perfectly calibrated probabilities and these two error costs is 1/(1+4)=0.2. A model fitted from finite data need not be perfectly calibrated, and validation estimates are noisy. The empirically chosen candidate may differ. Do not modify the threshold after seeing test errors and continue calling the test untouched.

Write a report with five sentences: the prediction task and data origin; the split and fitting procedure; the validation selection rule; the test counts, metrics, and baseline; and the limits of the claim. Follow the separation of fitting and evaluation described in scikit-learn’s cross-validation guide.

Transfer the procedure

For a real dataset, first establish permission to use it and define which features exist at prediction time. Identify related entities and time dependencies before choosing a split. Compare with a simple baseline and inspect enough examples to understand failures. Keep the experiment reproducible by recording source versions, exclusions, split identifiers, and parameters.

Completing this chapter means you ran the procedure, inspected its output, and can explain its boundaries. It is a foundation for further work. Search, probabilistic models, unsupervised learning, reinforcement learning, vision, and language each warrant deeper courses and larger projects. Start with the Berkeley CS 188 textbook and Dive into Deep Learning for those next steps.

Practice exercise

Run the chapter’s Python program locally. Before looking at the test output, record the chosen validation threshold and the cost rule. Add precision and recall with explicit handling for zero denominators. Compare with an always-negative baseline. Repeat the experiment with false-negative cost 1, and explain why a different threshold could be selected. Write a five-sentence report using the template in the chapter.

Compare with the worked solution

For an always-negative baseline, TP=FP=0, FN is the number of positives, and TN is the number of negatives. Recall is 0 when positives exist; precision is undefined because there are no positive predictions. Report undefined rather than claiming perfect precision. With false-negative cost 4, baseline cost is 4×FN; with cost 1 it is FN. Select each threshold using its own validation cost rule, then report the corresponding test counts without further tuning. A new threshold need not be lower or higher on every finite sample; compare the measured validation costs and document ties. The synthetic experiment checks the procedure, not real-world usefulness.

Check your understanding

You change the threshold after examining test mistakes. What is now true?

• The test is still untouched.
• That test set has become part of model selection.
• The model has proved generalization.
• The validation set is no longer needed.

Check the reasoning

Report the reuse honestly and obtain new held-out evidence for a final estimate.

An always-negative model makes no positive predictions. What should its precision be reported as?

• 100%.
• The model’s recall.
• Its accuracy.
• Undefined, with the zero denominator explained.

Check the reasoning

Precision is TP/(TP+FP). Here both numerator and denominator are zero; a software convention of zero is acceptable only if stated.

Learn AI curriculum