Calibration

A model is calibrated when its 70 percent calls win about 70 percent of the time, its 60 percent calls win about 60 percent of the time, and so on across every confidence band. Calibration is separate from accuracy: a model can be highly accurate and badly calibrated if it habitually overstates certainty.

The standard diagnostic is a reliability diagram. Bucket every graded prediction by stated confidence, plot the actual win rate for each bucket, and compare against the diagonal. Buckets above the line mean the model is under-confident; buckets below it mean it is over-confident.

When drift appears, the fix is a post-processing step such as Platt scaling or isotonic regression, applied to the raw model output rather than a change to the underlying features. AI4Gameday re-checks calibration continuously and tightens publish thresholds for a league before it ships more calls at the wrong confidence.