Expectation Consistent Approximate Inference
Manfred Opper and Ole Winther
Journal of Machine Learning Research
We propose a novel framework for approximations to
intractable probabilistic models which is based on a
free energy formulation. The
approximation can be understood from replacing an average
over the original intractable distribution with a tractable one.
It requires two tractable probability
distributions which are made consistent on a set of
moments and encode different features of the original intractable
distribution. In this way we are able to use
Gaussian approximations for models with discrete or bounded
variables which allow us to include non-trivial
correlations which are neglected in many other methods.
We test the framework on toy benchmark problems for
binary variables on fully connected graphs and 2D grids
and compare with other methods, such as loopy belief propagation.
Good performance is already achieved by using single nodes as tractable
substructures. Significant improvements are obtained when a spanning
tree is used instead.