PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

The Expected Performance Curve
Samy Bengio, Johnny Mariéthoz and Mikaela Keller
In: International Conference on Machine Learning (ICML) Workshop on ROC Analysis in Machine Learning, Bonn, Germany(2005).

Abstract

In several research domains concerned with classification tasks, curves like ROC are often used to assess the quality of a particular model or to compare two or more models with respect to various operating points. Researchers also often publish some statistics coming from the ROC, such as the so-called break-even point or equal error rate. The purpose of this paper is to first argue that these measures can be misleading in a machine learning context and should be used with care. Instead, we propose to use the Expected Performance Curves (EPC) which provide unbiased estimates of performance at various operating points. Furthermore, we show how to use adequately a non-parametric statistical test in order to produce EPCs with confidence intervals or assess the statistical significant difference between two models under various settings.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Lecture)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:1095
Deposited By:Samy Bengio
Deposited On:26 September 2005