Variational Bayes estimation of mixing coefficients
Bo Wang and Mike Titterington
In: Sheffild Workshop on Machine Learning, 7-10 Sept 2004, Sheffield, England.
We investigate theoretically some properties of variational Bayes approximations based on estimating the mixing coefficients of known densities. We show that, with probability 1 as the sample size n grows large, the iterative algorithm for the variational Bayes approximation converges locally to the maximum likelihood estimator at the rate of O(1/n). Moreover, the variational posterior distribution for the parameters is shown to be asymptotically normal with the same mean but a different covariance matrix compared with those for the maximum likelihood estimator. Furthermore, we prove that the covariance matrix from the variational Bayes approximation is small compared with that for the MLE, so that resulting interval estimates for the mixing weights will be unrealistically narrow.