Summary

This Task is a re-run with some extensions of Task 8 at SemEval 2014. We define broad-coverage semantic dependency parsing (SDP) as the task of recovering sentence-internal predicate–argument relationships for all content words, i.e. the semantic structure constituting the relational core of sentence meaning.  Target representations, thus, are semantic dependency graphs, as shown in our running example for the (Wall Street Journal) sentence:

A similar technique is almost impossible to apply to other crops, such as cotton, soybeans, and rice.

Here, ‘technique’ for example, is the argument of at least the determiner (as the quantificational locus), the intersective modifier ‘similar’, and the predicate ‘apply’.  Conversely, the predicative copula, infinitival ‘to’, and the vacuous preposition marking the deep object of ‘apply’ arguably have no semantic contribution of their own.  For general background on the 2014 variant and an overview of participating systems (and results), please see the Oepen et al. (2014).

The Task has three distinct target representations, dubbed DM, PAS, and PSD (renamed from what was PCEDT at SemEval 2014), representing different traditions of semantic annotation.  More detail on the linguistic ‘pedigree’ of these formats is available in the summary of target representations, and there is also an on-line search interface available to interactively explore these representations (like the initial release of the training data, this interface in early August 2014 still lacks semantic dependency graphs for languages other than English).

Training and testing data for the Task is distributed in a token-oriented, tabular file format; please see the data description for details.  A couple hundred sentences annotated in all three target representations are freely available as trial data.  As of early August 2014, the English portion of the training data is available for no-cost licensing through the LDC; please register for SemEval 2015 and subscribe to the SDP mailing list, to receive further information on access to the training data.

In comparison to SDP 2014, we will (a) re-run the same two tracks (‘closed’ and ‘open’) with the same training and testing data (Sections 00–21 of the venerable WSJ Corpus); and (b) augment the evaluation regime with two new metrics scoring complete predications, i.e. requiring the full set of argument dependencies for each predicate to be correct, and semantic frames, i.e. the combination of a complete predication with a specific predicate frame (or sense) identifier.  Furthermore, we will (c) add obligatory additional out-of-domain (i.e. non-WSJ) testing data, to gauge the resiliance of different approaches to variation in genre and domain.  Also, we will offer two new optional sub-tasks, viz. (d) cross-linguistic adaptation, i.e. training and testing data for additional languages (Czech and Chinese, for the PAS and PSD representations, respectively); and (e) predicate disambiguation, i.e. determining the correct argument frame (or sense) for each predicate (for DM and PSD, in English).  Finally, to further invest the contribution of syntactic analyses to the Task, we will (f) add a third ‘gold’ track, where idealized, gold-standard syntactic analyses will be provided in several formats.

At any point during the launch and execution of the Task, we will be very happy to receive feedback from prospective participants on the task design, data preparation, evaluation, or any other aspect of Task organization; please see the contact email address for the organizers to the right.

Motivation

Syntactic dependency parsing has seen great advances in the past decade, in part owing to relatively broad consensus on target representations, and in part reflecting the successful execution of a series of CoNLL shared tasks.  From this very active research area accurate and efficient syntactic parsers have developed for a wide range of natural languages.  However, the predominant target representation in dependency parsing to date are trees, in the formal sense that every node in the dependency graph is reachable from a distinguished root node by exactly one directed path.  This assumption is an essential prerequisite for both the parsing algorithms and the machine learning methods in state-of-the-art syntactic dependency parsers.  Unfortunately, this means that these parsers are ill-suited for producing meaning representations, i.e. moving from the analysis of grammatical structure to sentence semantics.  Even if syntactic parsing arguably can be limited to tree structures, this is obviously not the case in semantic analysis, where a node will often be the argument of multiple predicates (i.e. have more than one incoming arc), and it will often be desirable to leave some nodes unattached (with no incoming arcs), for semantically vacuous classes as, for example, particles, complementizers, or relative pronouns.

Thus, this Task seeks to stimulate the dependency parsing community to move towards more general graph processing, to thus enable semantic dependency parsing, i.e. a more direct analysis of ‘who did what to whom’.  Besides calling for node re-entrancies and partial connectivity (as evidenced in the sample target representations discussed above), an adequate representation of semantic dependencies will likely also exhibit higher degrees of non-projectivity than typical syntactic dependency trees.

Besides the relation to syntactic dependency parsing, the proposed task also has some overlap with Semantic Role Labeling (SRL).  In much previous work, however, target representations typically draw on resources like PropBank and NomBank, which are limited to argument identification and labeling for verbal and nominal predicates.  A plethora of semantic phenomena, e.g. negation and other scopal embedding, comparatives, possessives, various types of modification, and even conjunction, typically remain unanalyzed in SRL.  Thus, target representations are partial to a degree that can prohibit semantic downstream processing, for example inference-based techniques.  In this task, we require parsers to identify all semantic dependencies, i.e. compute a representation that integrates all content words in one structure.  Nevertheless, we anticipate that relatively straightforward adaptations of existing SRL approaches can be applied to yield broad-coverage semantic dependency parsing.

In recent years, we see beginning research into parsing with graph-structured representations, for example Sagae & Tsujii (2008), Das, et al. (2010), Jones, et al. (2013), Chiang, et al. (2013), and Henderson, et al. (2013).  However, some of these studies are purely theoretical, others limited to smaller, non-standard data sets.  We anticipate an increase in interest for this line of research, as well as emerging resources that can mature into broadly accepted target representations of semantic dependencies.

For these reasons, we expect that this SemEval 2015 Task will serve as a good vehicle to pull together, better understand, and make more widely accessible candidate target annotations, as well as to energize and synchronize emerging work on algorithms and statistical models for parsing into these types of more semantic representations.

Training and Testing Data

For English, we are aware of three independent annotations over the venerable WSJ text underlying the Penn Treebank (PTB) that have the formal and linguistic properties we are looking for:

• DM: The reduction of Minimal Recursion Semantics, available through the HPSG annotation of the WSJ text, into bi-lexical dependencies (Flickinger, et al., 2012).
• PAS: Predicate-Argument Structures extracted from another HPSG annotation of the PTB phrase structure trees (Miyao, et al., 2004).
• PSD: A reduction of the tectogrammatical analysis layer of the Prague Czech-English Dependency Treebank (Cinková, et al., 2009).

These resources constitute parallel semantic annotations over the same common text, but to date they have not been related to each other and, actually, have hardly been used for training and testing of data-driven analyzers.  As discussed to some degree in our 2014 Task Description, there are contentful differences between the DM, PAS, and PSD semantic annotations, and there is of course not one obvious (or even objective) truth.  For the English segment of this Task, we have synchronized these resources at the sentence and token levels (making sure they all annotate the exact same text), for approximately 750,000 annotated tokens in the WSJ domain.  More background on the linguistic characterization of these representations as well as on the task data format is available through separate pages.