Efficient Bounds for the softmax Function, Applications to Inference in Hybrid Models
In: NIPS 2007, 02-08 December 2007, Whistler, British Columbia, Canada.
We introduce the problem of variational inference in probabilistic model where a multinomial variable is defined conditionally to continous parents through a softmax function. Efficient Bayesian inference for this type of model is still an open problem due to the lack of efficient upper bound for the sum of exponentials. We propose three different bounds and compare them on simulations. As a real world example, the classification performances of the proposed bounding techniques for the Bayesian multinomial logistic regression are compared to approximations based on sampling. We finally discuss the pros and cons of "bounding" versus "approximating" the true posterior distribution.