Privacy, fairness, and data minimization

Prerequisites: Probability, classification errors, and access control.

Learning objectives

• Apply purpose limitation to a data flow.
• Compare group-level error rates.
• Explain why one fairness metric is incomplete.

Collection creates obligations and exposure

Privacy concerns how information about people is collected, used, shared, retained, and controlled. Security protects against unauthorized actions, but a securely implemented collection can still be excessive or contrary to people's expectations. Start with the purpose of each field and ask whether the task can be completed with less detail, shorter retention, or local computation.

A learning site may need completed lesson IDs to restore progress. It may not need raw keystrokes, precise location, or every draft answer indefinitely. Aggregate reporting can reduce exposure, but aggregation alone does not guarantee anonymity. Small groups, auxiliary data, and repeated releases can reveal individuals.

Follow the data lifecycle

Draw where data enters, which systems receive it, who can access it, and when it is deleted. Include logs, analytics, backups, exports, and third-party processors. A delete button that removes only the primary database row may leave other copies. State what deletion means and what retention constraints actually apply.

Collecting sensitive data “in case it becomes useful” creates a future risk without a current justification. Access controls, encryption, retention limits, and clear user choices address different parts of the problem. None replaces the decision about whether to collect the data at all.

Fairness requires a specified decision

Suppose a classifier flags learners for additional support. Group A has 100 learners who need support and the model identifies 80; group B has 50 and it identifies 30. Recall is 0.8 for A and 0.6 for B. Equal overall accuracy could hide this difference because the groups may have different sizes and base rates.

False positives and false negatives have consequences determined by the intervention. An unnecessary helpful invitation differs from an exclusionary decision. Identify affected stakeholders, label quality, historical bias, and the mechanism by which a prediction changes access or opportunity.

Different fairness criteria can conflict, especially when base rates differ and predictions are imperfect. Do not present one metric as a universal certificate. Report relevant group outcomes and uncertainty, inspect failure cases, and provide correction or appeal paths when decisions matter.

Governance is part of the system

Document the purpose, evidence, limitations, and accountable owner of a data-driven decision. Reevaluate when the population or intervention changes. A model can be technically accurate about a proxy while the proxy is unsuitable for the intended human outcome. Good engineering includes questioning the target and, when warranted, deciding not to automate the decision.

Practice problem

For a study platform, propose a minimal progress record and a retention policy for draft answers. Then compute recall when a group has 40 learners needing support and 28 are identified. State one question the recall number cannot answer.

Worked solution

A minimal progress record can include account ID, stable lesson ID, completion state, and necessary timestamps, with assessment details only when they support a stated learning purpose. Draft answers should have a clear user-facing retention and deletion behavior, including treatment of backups and logs. The appropriate duration depends on the actual purpose rather than a universal default. Recall is 28/40=0.7. It does not reveal false-positive burden, whether the labels are accurate, whether the support intervention helps, or whether the collection was justified.

Check your understanding

Does strong encryption justify collecting any available personal data?

• Yes, if the key is long.
• Yes, if users can sign in.
• No, collection still needs a purpose and proportionate scope.
• Only for educational sites.

Explanation

Privacy includes purpose and data use, not only protection against unauthorized access.

What is recall among people who need support?

• Identified people who need support divided by all people who need support.
• All correct predictions divided by all predictions.
• False positives divided by all alerts.
• The number of accounts in the database.

Explanation

Recall measures the fraction of actual positives identified.

Primary reading

• Introduction to Human-Computer Interaction: University of Washington CSE 440
• Web accessibility tutorials: W3C Web Accessibility Initiative
• Fairness and Machine Learning: Barocas, Hardt, and Narayanan

Open the study and save your work →