From Empirical Bayes to Leaving-One-Out
Empirical Bayes (EB) is a very appealing technique for tasks in which many outcomes of the population do not occur in the sample. In these tasks, it is necessary to estimate sparse probabilities and the EB is known to provide good estimates. However, EB estimates have two main drawbacks: they may be non-monotonic and under some circumstances they cannot be computed. This work presents a framework to constrain EB method by means of its equiva- lence with the leaving-one-out estimation. Two solutions are derived that amend the previous problems by applying two different sets of constraints: interval and monotonic. The typical application (for us) is language modelling.