PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

A Machine Learning based Approach to Evaluating Retrieval Systems
Huyen-Trang Vu and Patrick Gallinari
In: HLT-NAACL 2006, 5-7 June 2006, New York, USA.

Abstract

Test collections are essential to evaluate Information Retrieval systems. The relevance assessment set has been recognized as the key bottleneck in test collection building, especially on very large sized document collections. This paper addresses the problem of efficiently selecting documents to be included in the assessment set. We will show how machine learning techniques can fit this task. This leads to smaller pools than traditional round robin pooling, thus reduces significantly the manual assessment workload. Experimental results on TREC collections (http://trec.nist.gov) consistently demonstrate the effectiveness of our approach according to different evaluation criteria.

EPrint Type:Conference or Workshop Item (Oral)
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Information Retrieval & Textual Information Access
ID Code:2620
Deposited By:Huyen-Trang Vu
Deposited On:22 November 2006