PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Auxiliary Variational Information Maximization for Dimensionality Reduction
Felix Agakov and David Barber
In: PASCAL: Subspace, Latent Structure and Feature Selection techniques: Statistical and Optimisation perspectives Workshop 2005, 23-25 Feb 2005, Bohinj, Slovenia.


Mutual Information (MI) is a long studied measure of information content, and many attempts to apply it to feature extraction and stochastic coding have been made. However, in general MI is computationally intractable to evaluate, and most previous studies redefine the criterion in forms of approximations. Recently we described properties of a simple lower bound on MI, and discussed its links to some of the popular dimensionality reduction techniques. Here we introduce a richer family of auxiliary variational bounds on MI, which generalizes our previous approximations. Our specific focus then is on applying the bound to extracting informative lower-dimensional projections in the presence of irreducible Gaussian noise. We show that our method produces significantly tighter bounds than the well-known as-if Gaussian approximations of MI. We also show that the auxiliary variable method may help to significantly improve on reconstructions from noisy lower-dimensional projections. It may also be shown that our information-theoretic approach to stochastic dimensionality reduction generalizes self-supervised training in stochastic autoencoders.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Oral)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Learning/Statistics & Optimisation
ID Code:1107
Deposited By:Felix Agakov
Deposited On:29 September 2005