FREE AI & machine learning LESSON · AI & machine learning
Operate the model as a system
Calibration, drift, guardrails, and feedback
Deployment changes the data-generating process.
Once predictions affect actions, the model can alter which outcomes are observed and which examples return as labels. Input distributions drift, policies change, users adapt, and retraining pipelines introduce new code and data. Monitor the decision path: data quality, score distributions, calibration, slice outcomes, intervention rates, and delayed ground truth.
A model is one component in a feedback system with people, policy, software, and time.
Feedback can erase the counterfactual.
If high-risk cases always receive an intervention, their natural outcome becomes unobservable. Training on observed outcomes may teach the model that treated cases were safe, or encode previous policy as ground truth. Preserve treatment information, use experiments where ethical, and distinguish prediction from causal effect.
What happened after an action is not automatically what would have happened without it.