PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Estimation of Information Theoretic Measures for Continuous Random Variables
Fernando Perez-Cruz
In: Neural Information Processing Systems, 8-13 Dec 2008, Vancouver, Canada.

Abstract

We analyze the estimation of information theoretic measures of continuous random variables such as: differential entropy, mutual information or Kullback- Leibler divergence. The objective of this paper is two-fold. First, we prove that the information theoretic measure estimates using the k-nearest-neighbor density estimation with fixed k converge almost surely, even though the k-nearest-neighbor density estimation with fixed k does not converge to its true measure. Second, we show that the information theoretic measure estimates do not converge for k growing linearly with the number of samples. Nevertheless, these nonconvergent estimates can be used for solving the two-sample problem and assessing if two random variables are independent. We show that the two-sample and independence tests based on these nonconvergent estimates compare favorably with the maximum mean discrepancy test and the Hilbert Schmidt independence criterion.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Paper)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:4908
Deposited By:Fernando Perez-Cruz
Deposited On:24 March 2009