|
Generalization to Unseen Cases
Teemu Roos, Peter Grünwald, Petri Myllymäki and Henry Tirri
In: NIPS 2005, 5-11 Dec 2005, Vancouver, Canada.
AbstractWe analyse classification error on unseen cases, i.e. cases that are different from those in the training set. Unlike standard generalization error, this {\em off-training set error\/} may differ significantly from the empirical error with high probability even with large sample sizes. We derive a data-dependent bound on the difference between off-training set and standard generalization error. Our result is based on a new bound on the missing mass, which for small samples is stronger than existing bounds based on Good-Turing estimation. As we demonstrate on the UCI data sets, our bound gives nontrivial generalization guarantees in many practical cases. In light of these results, we show that certain claims made in the No Free Lunch literature are overly pessimistic. [Edit]
|
