Crowd-sourced Review

Crowd-sourced review of test data: from Jan 5 to Jan 10 [OPTIONAL participation]

Participants can opt to participate also in the social review of test data. Organizers will release the test questions and key answers, together which each participant answers. Participants can review their system answers and compare them with the key ones and if they think the key is wrong they can send a review claim to organizers.

Organizers will go over all the claims and decide which ones are correct and will update the test key. If finally the test data is updated, organizers will reevaluate systems against the reviewed testset and publish the final results.

Hope to count on your participation on this task that will ensure and perhaps improve the test data quality.

Contact Info


  • Hector Llorens
  • Nate Chambers
  • Naushad UzZaman
  • Nasrin Mostafazadeh
  • James Allen
  • James Pustejovsky

email :

Other Info


  • 2015-03-09 UPDATED results of QA-TempEval
  • 2015-01-15 Results of QA-TempEval
  • Fill in SemEval registration form to participate
  • 2014-11-04 evaluation period starts on December 15
  • 2014-12-01 crowd-sourced test creation: OPEN (from Dec 5 to Jan 10)
  • 2014-11-04 A new domain will be included: informal blogs
  • 2014-08-26 PDF detailed task description
  • 2014-06-23 Train QA data released (data&tools)
  • 2014-06-14 Dev QA data released (data&tools)
  • 2014-06-02 TimeML QA system released (data&tools)
  • 2014-05-30 TimeML data available (TempEval-3 format) (data&tools)