Some Discriminant-based PAC Algorithms
There is a more recent version of this eprint available. Click here to view it.
A common approach in pattern classification is to estimate, for each label class, a probability distribution that generated the observations having that label, and then to label a new unlabeled observation by applying the Bayes classifier to the estimated distributions. That approach provides more useful information about a data point that just a class label; it also provides estimates of the conditional distribution of class labels, in situations where there is class overlap. We would like to know whether it is harder to classify data via this approach, than by techniques that may consider all data of distinct class labels together. In this paper we make that question precise by considering it in the context of PAC learnability, and ask whether there are any PAC learnable concept classes that are not learnable in a restriction that corresponds to learning via estimates of the distributions. We do not have an example of a concept class that actually distinguishes the restricted setting from general PAC learnability. Instead we have found some surprising positive results for the restricted setting, which allow us to conclude that our restriction is not more demanding than various other well-known restrictions of PAC learnability. For example the restriction is not a subset of learnability with one-sided error, or in the presence of misclassification noise, or in the mistake-bound (query) model. Those last two results assume respectively the noisy parity assumption and the existence of one-way functions. The learnability of simple monomials remains an open problem within the restricted setting, assuming no knowledge of the distribution over inputs. We give an algorithm for learning monomials over input vectors generated by an unknown product distribution.
Available Versions of this Item