PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Trading value and information in MDPs
Jonathan Rubin, Ohad Shamir and Naftali Tishby
In: Decision Making with Imperfect Decision Makers (2011) Springer .

Abstract

Interactions between an organism and its environment are commonly treated in the framework of Markov Decision Processes (MDP). While standard MDP is aimed at maximizing expected future rewards (value), the circular flow of information between the agent and its environment is generally ignored. In partic- ular, the information gained from the environment by means of perception and the information involved in the process of action selection are not treated in the standard MDP setting. In this paper, we focus on the control information and show how it can be combined with the reward measure in a unified way. Both of these measures satisfy the familiar Bellman recursive equations, and their linear combination (the free-energy) provides an interesting new optimization criterion. The tradeoff between value and information, explored using our INFO-RL algorithm, provides a principled justification for stochastic (soft) policies. We use computational learn- ing theory to show that these optimal policies are also robust to uncertainties in the reward values.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Book Section
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Computational, Information-Theoretic Learning with Statistics
Learning/Statistics & Optimisation
Theory & Algorithms
ID Code:9320
Deposited By:Amir Globerson
Deposited On:16 March 2012