Overview of Morpho Challenge in CLEF 2007
Mikko Kurimo, Mathias Creutz and Ville Turunen
In: Morpho Challenge Workshop at CLEF 2007, 19-21 Sep 2007, Budapest, Hungary.
Morpho Challenge 2007 contained an evaluation of unsupervised morpheme analysis
algorithms using information retrieval experiments utilizing data available in CLEF.
The objective of the challenge was to design statistical machine learning algorithms
that discover which morphemes (smallest individually meaningful units of language)
words consist of. Ideally, these are basic vocabulary units suitable for different tasks,
such as text understanding, machine translation, information retrieval, and statistical
language modeling The evaluation of the submitted morpheme analysis was performed
by two complementary ways: Competition 1: The proposed morpheme analyses were
compared to a linguistic morpheme analysis gold standard by matching the morphemesharing
word pairs. Competition 2: Information retrieval (IR) experiments were performed,
where the words in the documents and queries were replaced by their proposed
morpheme representations and the search was based on morphemes instead of words.
This paper provides an overview of the IR evaluation. The IR evaluations were provided
for Finnish, German, and English and participants were encouraged to apply
their algorithm to all of them. The organizers performed the IR experiments using
the queries, texts, and relevance judgments available in CLEF forum and morpheme
analysis methods submitted by the challenge participants. The results show that the
morpheme analysis has a significant effect in IR performance in all languages, and that
the performance of the best unsupervised methods can be superior to the supervised
reference methods. The challenge was part of the EU Network of Excellence PASCAL
Challenge Program and organized in collaboration with CLEF.