PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

A statistical approach to crosslingual natural language tasks
David Pinto, Jorge Civera, Alberto Barrón, Alfons Juan and Paolo Rosso
Journal of Algorithms 2008. ISSN 0196-6774

Abstract

The existence of huge volumes of documents written in multiple languages on Internet leads to investigate novel algorithmic approaches to deal with information of this kind. However, most crosslingual natural language processing (NLP) tasks consider a decoupled approach in which monolingual NLP techniques are applied along with an independent translation process. This two-step approach is too sensitive to translation errors, and in general to the accumulative effect of errors. To solve this problem, we propose to use a direct probabilistic crosslingual NLP system which integrates both steps, translation and the specific NLP task, into a single one. In order to perform this integrated approach to crosslingual tasks, we propose to use the statistical IBM 1 word alignment model (M1). The M1 model may show a non-monotonic behaviour when aligning words from a sentence in a source language to words from another sentence in a different, target language. This is the case of languages with different word order. In English, for instance, adjetives appear before nouns, whereas in Spanish it is exactly the opposite. The successful experimental results reported in three different tasks – text classification, information retrieval and plagiarism analysis– highlight the benefits of the statistical integrated approach proposed in this work.

EPrint Type:Article
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Learning/Statistics & Optimisation
Natural Language Processing
Information Retrieval & Textual Information Access
ID Code:4559
Deposited By:Alfons Juan
Deposited On:24 March 2009