Efficient reinforcement learning in parametrized models: Discrete parameter case.
Kirill Dyagilev, Shie Mannor and Nahum Shimkin
Lecture Notes In Artificial Intelligence. Recent Advances in Reinforcement Learning: 8th European Workshop, Ewrl 2008, Villeneuve D'ascq, France, June 30-July 3, 2008, Revised and Selected Papers pp. 41-54, 2009.

Abstract

We consider reinforcement learning in the parametrized setup, where the model is known to belong to a finite set of Markov Decision Processes (MDPs) under the discounted return criteria. We propose an on-line algorithm for learning in such Parametrized models, the Parameter Elimination (PEL) algorithm, and analyze its performance in terms of the total mistake bound criterion. The algorithm relies on Wald's sequential probability ratio test to eliminate unlikely parameters, and uses an optimistic policy for effective exploration. We establish that, with high probability, the total mistake bound for the algorithm is linear (up to a logarithmic term) in the size |\Theta| of the parameter space, independently of the cardinality of the state and action spaces. We further demonstrate that much better dependence on |\Theta| is possible, depending on the specific information structure of the problem.

[Edit]