Bayes and Tukey meet at the Center Point
Ran Gilad-Bachrach, Amir Navot and Naftali Tishby
Proceedings of the ACM Conference on Comptational Learning Theory
The Bayes classifier achieves the minimal error rate by constructing a weighted majority over all concepts in the concept class. The "Bayes Point" uses the single concept in the class which has the minimal error. This way, the Bayes Point avoids some of the deficiencies of the Bayes classifier. We prove a bound on the generalization error for Bayes Point Machines when learning linear classifiers and show that it is at most ~1.71 times the generalization error of the Bayes classifier, independent of the input dimension and length of training. We show that when learning linear classifiers, the Bayes Point is almost identical to the Tukey Median and Center Point. We extend these definitions beyond linear classifiers and define the Bayes Depth of a classifier. We prove generalziation bound in terms of this new definition. Finally, we provide a new concentration of measure inequality for multivariate random variables to the Tukey Median.