Brier score

The Brier score takes each prediction, squares the difference between the stated probability and the actual result (1 for a win, 0 for a loss), and averages across all predictions. A perfect forecaster scores 0. Always guessing 50 percent scores 0.25.

It rewards being both accurate and appropriately confident. A confident wrong call is punished far more than a hesitant wrong call, which is exactly the incentive an honest forecasting system should face.

Brier score is best read alongside a reliability diagram. The score tells you how good the forecasts are overall; the diagram tells you where the model is over- or under-confident.