PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Model-based, multimodal interaction in document browsing
P Eslambolchilar and Roderick Murray-Smith
In: 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms (MLMI'06), May 2006, Washington DC, USA.


In this paper we introduce a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in browsing a document, by adapting the dynamics of navigation and the visual feedback (using a focus-in-context (F+C) method) to support the current inferred task. We also demonstrate non-speech audio feedback, based on a language model. We argue that to design interaction we need models of key aspects of the process, here for example, we need models for the dynamic system, language model and soni¯cation. We show how the user's intention is coupled to the visualization technique via the dynamic model, and how the focus-in- context method couples details in context to audio samples via the language identi¯cation system. We present probabilistic audio feedback as an example of a multimodal approach to sensing di®erent languages in a multilingual text. This general approach is well suited to mobile and wearable applications, and shared displays.

PDF - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Conference or Workshop Item (Invited Talk)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:User Modelling for Computer Human Interaction
Multimodal Integration
Information Retrieval & Textual Information Access
ID Code:2461
Deposited By:Roderick Murray-Smith
Deposited On:22 November 2006