Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation
Roy Schwartz, Omri Abend, Roi Reichart and Ari Rappoport
In: ACL 2011(2011).
Dependency parsing is a central NLP task. In
this paper we show that the common evaluation
for unsupervised dependency parsing
is highly sensitive to problematic annotations.
We show that for three leading unsupervised
parsers (Klein and Manning, 2004; Cohen and
Smith, 2009; Spitkovsky et al., 2010a), a small
set of parameters can be found whose modification
yields a significant improvement in
standard evaluation measures. These parameters
correspond to local cases where no linguistic
consensus exists as to the proper gold
annotation. Therefore, the standard evaluation
does not provide a true indication of algorithm
quality. We present a new measure, Neutral
Edge Direction (NED), and show that it greatly
reduces this undesired phenomenon.