PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Fast parallel estimation of high dimensional information theoretical quantities with low dimensional random projection ensembles
Zoltan Szabo and András Lorincz
In: 8th International Conference on Independent Component Analysis and Signal Separation (ICA), 15-18 Mar 2009, Paraty, Brazil.

This is the latest version of this eprint.

Abstract

The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task.We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method.We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.

PDF - PASCAL Members only - Requires Adobe Acrobat Reader or other PDF viewer.
PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Paper)
Additional Information:http://dx.doi.org/10.1007/978-3-642-00599-2_19
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:9529
Deposited By:Zoltan Szabo
Deposited On:29 May 2012

Available Versions of this Item