The human override rate, the richest indicator in a decision system
The percentage of files where the analyst does not follow the recommendation says three things at once. It is the only indicator that measures the model, the data and the organisation simultaneously.
An indicator few institutions record
Most scoring systems measure default rate and volume handled. Almost none systematically measures the gap between what the system recommended and what the human decided — nor, above all, the reason for that gap.
That is a pity, because the gap is the only signal that crosses all three layers of a system: model quality, data quality, and organisational buy-in.
Three readings of one number
A high gap concentrated on one segment signals a calibration flaw: the model is systematically wrong about a population, and analysts correct it by hand. That is modelling information, available before the portfolio shows it.
A high gap that is dispersed, and justified by elements absent from the file, signals a blind spot in the data: the analyst knows something the system cannot see. That is data engineering information, and often the most profitable to act on.
A high gap with no consistent reason signals a buy-in or training problem. That is change management information — and the only case where the answer is not technical.
Why written grounds must be required
A gap with no recorded reason is information lost. Requiring written grounds slows the analyst by seconds and makes the indicator statistically usable.
The usual objection is that analysts will fill the field mechanically. That is true if the reason is a free-text field with no consequence. It is far less true when reasons are categorised, aggregated, and returned to the risk department as analysis — because analysts then see that their input is used.
Why a zero override rate should worry you
A gap of zero is not a success. It means either that analysts apply the recommendation without examining it, or that they have no latitude to depart from it. In both cases, human supervision has become a formality.
A healthy system produces a small but non-zero gap, justified, whose reasons shift as the model and the data improve. It is that movement, more than the absolute level, that shows the feedback loop is working.
Sources
- NEXCLARA functional specifications — VISION module, decision quality indicators.
- WAMU Banking Commission — Circular no. 04-2017/CB/C, internal control requirements.