PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Lessons learned in the challenge: making predictions and scoring them
Jukka Kohonen and Jukka Suomela
In: Evaluating Predictive Uncertainty, Textual Entailment and Object Recognition Systems: Selected Proceedings of the 1st PASCAL Machine Learning Challenges Workshop Springer Lecture Notes in Artificial Intelligence . (2005) Springer-Verlag .


In this paper we present lessons learned in the Evaluating Predictive Uncertainty Challenge. We describe the methods we used in regression challenges, including our winning method for the Outaouais data set. We then turn our attention to the more general problem of scoring in probabilistic machine learning challenges. It is widely accepted that scoring rules should be proper in the sense that the true generative distribution has the best expected score; we note that while this is useful, it does not guarantee finding the best methods for practical machine learning tasks. We point out some problems in local scoring rules such as the negative logarithm of predictive density (NLPD), and illustrate with examples that many of these problems can be avoided by a distance-sensitive rule such as the continuous ranked probability score (CRPS).

PDF - PASCAL Members only - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Book Section
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
ID Code:1573
Deposited By:Jukka Kohonen
Deposited On:28 November 2005