The Expected Performance Curve
Samy Bengio, Johnny Mariéthoz and Mikaela Keller
In: International Conference on Machine Learning (ICML) Workshop on ROC Analysis in Machine Learning, Bonn, Germany(2005).
In several research domains concerned with classification tasks,
curves like ROC
are often used to assess the quality of a particular model or to compare two
or more models with respect to various operating points.
Researchers also often publish some statistics coming
from the ROC, such as the so-called break-even point or equal
error rate. The purpose of this paper is to first argue that these measures
can be misleading in a machine learning context
and should be used with care. Instead, we propose
to use the Expected Performance Curves (EPC) which provide unbiased
estimates of performance at various operating points.
Furthermore, we show how to use adequately a
non-parametric statistical test in order to produce EPCs with
confidence intervals or assess the statistical significant difference
between two models under various settings.