Tool
Calibration auditor
A confidence score means nothing until it has been measured against outcomes. Give this a CSV with a predicted probability and an observed outcome, and it will tell you whether your system's stated confidence matches reality.
or drop one here —
File format
Two columns. The first a predicted probability between 0 and 1 (percentages work too), the second the observed outcome as 1/0, true/false, or yes/no. A header row is optional — if present, columns named like prediction, score, or confidence and outcome, actual, or label are detected automatically. Minimum 20 rows.
Everything is computed in your browser. The file is never uploaded and no network request is made.
Contact
If you are working on this problem, I would like to hear about it.
These are the questions I spend my time on. If you are building or buying a system that makes consequential calls and cannot explain them, write to me.
phil@philpierce.com