Joint analysis of audio and text data in CASAM
Sergios Petridis, Katerina Papantoniou, Theodoros Giannakopoulos, Georgios Paliouras, Miltiadis Koutsokeras, Ilias Zavitsanos, George Tsatsaronis, Georgios Akrivas, Basilis Gatos, Kostas Ntirogiannis and Stavros Perantonis
In: SAMT 2010, 1-3 Dec. 2010, Saarbrücken, Germany.
This paper presents the approach used to extract information from multimedia in the context of the Computer-Aided Semantic Annotation of Multimedia (CASAM) system. In particular, we rst describe from a system' s perspective the relevant component of the system, named Knowledge Driven Multimedia Analysis (KDMA) component. We then focus on a particular methodology that allows to improve detection of information found in audio stream of a document, using information found in related text data, provided either as auxiliary sources, speech or user annotations. The methodology is based on separately analysing each medium and then learn a mapping among concepts found in audio and text. This mapping is later used to propose priors for audio classes at the document level and use them to adapt the audio classes posteriors. The evaluation results of the described analysis methods on a multimedia news items corpus demonstrate the usefulness of the approach.