SemEval-2016 Task 8

Overview:

Abstract Meaning Representation (AMR) is a compact, readable, whole-sentence semantic annotation. Annotation components include entity identification and typing, PropBank semantic roles, individual entities playing multiple roles, entity grounding via wikification, as well as treatments of modality, negation, etc.

Here is an example AMR for the sentence “The London emergency services said that altogether 11 people had been sent to hospital for treatment due to minor wounds.”

(s / say-01
      :ARG0 (s2 / service
            :mod (e / emergency)
            :location (c / city :wiki ‘‘London’’
                  :name (n / name :op1 ‘‘London’’)))
      :ARG1 (s3 / send-01
            :ARG1 (p / person :quant 11)
            :ARG2 (h / hospital)
            :mod (a / altogether)
            :purpose (t / treat-03
                  :ARG1 p
                  :ARG2 (w / wound-01
                        :ARG1 p
                        :mod (m / minor)))))

Note the inclusion of PropBank semantic frames (‘say-01’, ‘send-01’, ‘treat-03’, ‘wound-01’), grounding via wikification (‘London’), and multiple roles played by an entity (e.g. ‘11 people’ are the ARG1 of send-01, the ARG1 of treat-03, and the ARG1 of wound-01).

With the recent public release of a sizeable corpus of English/AMR pairs (LDC2014T12), there has been substantial interest in creating parsers to recover this formalism from plain text.  Several parsers have already been released (see reference list below) and more may be on their way soon.  It seems an appropriate time to conduct a carefully guided shared task so that this nascent community may cleanly evaluate their various approaches side by side under controlled scenarios. 

Rules:

Participants will be provided with parallel English-AMR training data. They will have to parse new English data and return the obtained AMRs. Participants may use any resources at their disposal (but may not hand-annotate the blind data or hire other human beings to hand-annotate the blind data). The SemEval trophy goes to the system with the highest Smatch score.

Existing AMR Parsers: (send email to jonmay@isi.edu if yours is missing and you want a citation)

• A Discriminative Graph-Based Parser for the Abstract Meaning Representation (ACL 2014)
  • Jeffrey Flanigan Sam Thomson Jaime Carbonell Chris Dyer Noah A. Smith
  • http://www.aclweb.org/anthology/P/P14/P14-1134.pdf
• A Transition-based Algorithm for AMR Parsing (NAACL 2015)
  • Chuan Wang; Nianwen Xue; Sameer Pradhan
  • http://www.aclweb.org/anthology/N/N15/N15-1040.pdf
  • Updated at ACL 2015: http://www.aclweb.org/anthology/P/P15/P15-2141.pdf
• An AMR parser for English, French, German, Spanish and Japanese and a new AMR-annotated corpus (NAACL 2015)
  • Lucy Vanderwende; Arul Menezes; Chris Quirk
  • http://www.aclweb.org/anthology/N/N15/N15-3006.pdf
• A Synchronous Hyperedge Replacement Grammar based approach for AMR parsing (CoNLL 2015)
  • Xiaochang Peng; Linfeng Song; Daniel Gildea
  • http://www.aclweb.org/anthology/K/K15/K15-1004.pdf
• Parsing English into Abstract Meaning Representation Using Syntax-Based Machine Translation (EMNLP 2015)
  • Michael Pust; Ulf Hermjakob; Kevin Knight; Daniel Marcu; Jonathan May
  • http://www.aclweb.org/anthology/D/D15/D15-1136.pdf
• Broad-coverage CCG Semantic Parsing with AMR (EMNLP 2015)
  • Yoav Artzi; Kenton Lee; Luke Zettlemoyer
  • http://www.aclweb.org/anthology/D/D15/D15-1198.pdf