PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Temporal visual cues aid speech recognition
X Zhou, L. Ross, Jan Larsen and L Parra
In: 7th Annual Meeting of the International Multisensory Research Forum(2006).


BACKGROUND: It is well known that under noisy conditions, viewing a speaker's articulatory movement aids the recognition of spoken words. Conventionally it is thought that the visual input disambiguates otherwise confusing auditory input. HYPOTHESIS: In contrast we hypothesize that it is the temporal synchronicity of the visual input that aids parsing of the auditory stream. More specifically, we expected that purely temporal information, which does not convey information such as place of articulation may facility word recognition. METHODS: To test this prediction we used temporal features of audio to generate an artificial talking-face video and measured word recognition performance on simple monosyllabic words. RESULTS: When presenting words together with the artificial video we find that word recognition is improved over purely auditory presentation. The effect is significant (p<0.01) for SNR at or above -12dB noise. For lower SNR the visual temporal information does not improve recognition confirming that our visual input does not contain useful lip-reading information in itself. CONCLUSION: Thus, we argue that temporal information is used in addition to articulatory features. This finding supports the notion that synchronous visual input aids auditory processing at an early parsing stage.

EPrint Type:Conference or Workshop Item (Talk)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:User Modelling for Computer Human Interaction
Multimodal Integration
ID Code:2817
Deposited By:Tue Lehn-Schiøler
Deposited On:22 November 2006