PASCAL - Pattern Analysis, Statistical Modelling and Computational Learning

Analysis of the IJCNN 2007 Agnostic Learning vs. Prior Knowledge Challenge
Isabelle Guyon, Amir Saffari, Gideon Dror and Gavin Cawley
Neural Networks 2007.

Abstract

We organized a challenge for IJCNN 2007 to assess the added value of prior domain knowledge in machine learning. Most commercial data mining programs accept data pre-formatted in the form of a table, with each example being encoded as a linear feature vector. Is it worth spending time incorporating domain knowledge in feature construction or algorithm design or can off-the-shelf programs working directly on simple low-level features do better than skilled data analysts? To answer these questions, we formatted five datasets using two data representations. The participants to the “prior knowledge” track used the raw data, with full knowledge of the meaning of the data representation. Conversely, the participants to the “agnostic learning” track used a pre-formatted data table, with no knowledge of the identity of the features. The results indicate that black-box methods using relatively unsophisticated features work quite well and rapidly approach the best attainable performances. The winners on the prior knowledge track used feature extraction strategies yielding a large number of low-level features. Incorporating prior knowledge in the form of generic coding/smoothing methods to exploit regularities in data is beneficial, but incorporating actual domain knowledge in feature construction is very time consuming and seldom leads to significant improvements. The AL vs. PK challenge web site remains open for post-challenge submissions: http://www.agnostic.inf.ethz.ch/ .

PDF - PASCAL Members only - Requires Adobe Acrobat Reader or other PDF viewer.
EPrint Type:Article
Additional Information:An on-line report of the challenge is available at: http://clopinet.com/isabelle/Projects/agnostic/Results.html
Project Keyword:Project Keyword UNSPECIFIED
Subjects:Theory & Algorithms
ID Code:4039
Deposited By:Isabelle Guyon
Deposited On:25 February 2008