SDP 2015: Broad-Coverage Semantic Dependency Parsing

Summary

This Task is a re-run with some extensions of Task 8 at SemEval 2014. We define broad-coverage semantic dependency parsing (SDP) as the task of recovering sentence-internal predicate–argument relationships for all content words, i.e. the semantic structure constituting the relational core of sentence meaning.  Target representations, thus, are semantic dependency graphs, as shown in our running example for the (Wall Street Journal) sentence:

A similar technique is almost impossible to apply to other crops, such as cotton, soybeans, and rice.

Here, ‘technique’ for example, is the argument of at least the determiner (as the quantificational locus), the intersective modifier ‘similar’, and the predicate ‘apply’.  Conversely, the predicative copula, infinitival ‘to’, and the vacuous preposition marking the deep object of ‘apply’ arguably have no semantic contribution of their own.  For general background on the 2014 variant and an overview of participating systems (and results), please see the Oepen et al. (2014).

The Task has three distinct target representations, dubbed DM, PAS, and PSD (renamed from what was PCEDT at SemEval 2014), representing different traditions of semantic annotation.  More detail on the linguistic ‘pedigree’ of these formats is available in the summary of target representations, and there is also an on-line search interface available to interactively explore these representations (like the initial release of the training data, this interface in early August 2014 still lacks semantic dependency graphs for languages other than English).

Training and testing data for the Task is distributed in a token-oriented, tabular file format; please see the data description for details.  A couple hundred sentences annotated in all three target representations are freely available as trial data.  As of early August 2014, the English portion of the training data is available for no-cost licensing through the LDC; please register for SemEval 2015 and subscribe to the SDP mailing list, to receive further information on access to the training data.

In comparison to SDP 2014, we will (a) re-run the same two tracks (‘closed’ and ‘open’) with the same training and testing data (Sections 00–21 of the venerable WSJ Corpus); and (b) augment the evaluation regime with two new metrics scoring complete predications, i.e. requiring the full set of argument dependencies for each predicate to be correct, and semantic frames, i.e. the combination of a complete predication with a specific predicate frame (or sense) identifier.  Furthermore, we will (c) add obligatory additional out-of-domain (i.e. non-WSJ) testing data, to gauge the resiliance of different approaches to variation in genre and domain.  Also, we will offer two new optional sub-tasks, viz. (d) cross-linguistic adaptation, i.e. training and testing data for additional languages (Czech and Chinese, for the PAS and PSD representations, respectively); and (e) predicate disambiguation, i.e. determining the correct argument frame (or sense) for each predicate (for DM and PSD, in English).  Finally, to further invest the contribution of syntactic analyses to the Task, we will (f) add a third ‘gold’ track, where idealized, gold-standard syntactic analyses will be provided in several formats.

At any point during the launch and execution of the Task, we will be very happy to receive feedback from prospective participants on the task design, data preparation, evaluation, or any other aspect of Task organization; please see the contact email address for the organizers to the right.

Motivation

Syntactic dependency parsing has seen great advances in the past decade, in part owing to relatively broad consensus on target representations, and in part reflecting the successful execution of a series of CoNLL shared tasks.  From this very active research area accurate and efficient syntactic parsers have developed for a wide range of natural languages.  However, the predominant target representation in dependency parsing to date are trees, in the formal sense that every node in the dependency graph is reachable from a distinguished root node by exactly one directed path.  This assumption is an essential prerequisite for both the parsing algorithms and the machine learning methods in state-of-the-art syntactic dependency parsers.  Unfortunately, this means that these parsers are ill-suited for producing meaning representations, i.e. moving from the analysis of grammatical structure to sentence semantics.  Even if syntactic parsing arguably can be limited to tree structures, this is obviously not the case in semantic analysis, where a node will often be the argument of multiple predicates (i.e. have more than one incoming arc), and it will often be desirable to leave some nodes unattached (with no incoming arcs), for semantically vacuous classes as, for example, particles, complementizers, or relative pronouns.

Thus, this Task seeks to stimulate the dependency parsing community to move towards more general graph processing, to thus enable semantic dependency parsing, i.e. a more direct analysis of ‘who did what to whom’.  Besides calling for node re-entrancies and partial connectivity (as evidenced in the sample target representations discussed above), an adequate representation of semantic dependencies will likely also exhibit higher degrees of non-projectivity than typical syntactic dependency trees.

Besides the relation to syntactic dependency parsing, the proposed task also has some overlap with Semantic Role Labeling (SRL).  In much previous work, however, target representations typically draw on resources like PropBank and NomBank, which are limited to argument identification and labeling for verbal and nominal predicates.  A plethora of semantic phenomena, e.g. negation and other scopal embedding, comparatives, possessives, various types of modification, and even conjunction, typically remain unanalyzed in SRL.  Thus, target representations are partial to a degree that can prohibit semantic downstream processing, for example inference-based techniques.  In this task, we require parsers to identify all semantic dependencies, i.e. compute a representation that integrates all content words in one structure.  Nevertheless, we anticipate that relatively straightforward adaptations of existing SRL approaches can be applied to yield broad-coverage semantic dependency parsing.

In recent years, we see beginning research into parsing with graph-structured representations, for example Sagae & Tsujii (2008), Das, et al. (2010), Jones, et al. (2013), Chiang, et al. (2013), and Henderson, et al. (2013).  However, some of these studies are purely theoretical, others limited to smaller, non-standard data sets.  We anticipate an increase in interest for this line of research, as well as emerging resources that can mature into broadly accepted target representations of semantic dependencies.

For these reasons, we expect that this SemEval 2015 Task will serve as a good vehicle to pull together, better understand, and make more widely accessible candidate target annotations, as well as to energize and synchronize emerging work on algorithms and statistical models for parsing into these types of more semantic representations.

Training and Testing Data

For English, we are aware of three independent annotations over the venerable WSJ text underlying the Penn Treebank (PTB) that have the formal and linguistic properties we are looking for:

These resources constitute parallel semantic annotations over the same common text, but to date they have not been related to each other and, actually, have hardly been used for training and testing of data-driven analyzers.  As discussed to some degree in our 2014 Task Description, there are contentful differences between the DM, PAS, and PSD semantic annotations, and there is of course not one obvious (or even objective) truth.  For the English segment of this Task, we have synchronized these resources at the sentence and token levels (making sure they all annotate the exact same text), for approximately 750,000 annotated tokens in the WSJ domain.  More background on the linguistic characterization of these representations as well as on the task data format is available through separate pages.

Contact Info

Organizers

  • Dan Flickinger
  • Jan Hajič
  • Angelina Ivanova
  • Marco Kuhlmann
  • Yusuke Miyao
  • Stephan Oepen
  • Daniel Zeman

sdp-organizers@emmtee.net

Other Info

Announcements

[06-feb-15] Final evaluation results for the task are now available; we are grateful to all (six) participating teams.

[08-dec-15] The evaluation period is nearing completion; we have purged inactive subscribers from the task-specific mailing list and sent out important information on the submssion of system outputs for evaluation to the list; if you have not received this email but are actually preparing a system submission, please contact the organizers immediately.

[17-dec-14] We are about to enter the evaluation phase, but recall that the closing date has been extended to Thursday, January 15, 2015. We have sent important instructions on how to participate in the evaluation to the task-specific mailing list; if you plan on submitting system results to this task but have not seen these instructions, please make contact with the organizers immediately.

[22-nov-14] English ‘companion’ syntactic analyses in various dependency formats are now available, for use in the open and gold tracks.

[20-nov-14] We have completed the production of cross-lingual training data: some 31,000 PAS graphs for Chinese and some 42,000 PSD graphs for Czech. At the same time, we have prepared an update of the English training data, with somewhat better coverage and a few improved analyses in DM, as well as with additional re-entrancies (corresponding to grammatical control relations) in PSD. The data is available for download as Version 1.1 from the LDC. Owing to the delayed availability of the cross-lingual data, we have moved the closing date for the evaluation period to mid-January 2015.

[14-nov-14] An update to the SDP toolkit (now hosted at GitHub) is available, implementing the additional evaluation metrics ‘complete predicates’ and ‘semantic frames’.

[05-aug-14] We are (finally) ready to officially ‘launch’ SDP 2015: the training data is now available for distribution through the LDC; please register for SemEval 2015 Task 18, and within a day (or so) we will be in touch about data licensing and access information.

[03-aug-14] Regrettably, we are running late in making available the training data and technical details of the 2015 task setup; please watch this page for updates over the next couple of days!

[01-jun-14] We have started to populate the task web pages, including some speculative information on extensions (compared to the 2014 variant of the task) that we are still discussing. A first sample of trial data is available for public download.