Crowd-sourced Review
Crowd-sourced review of test data: from Jan 5 to Jan 10 [OPTIONAL participation]
Participants can opt to participate also in the social review of test data. Organizers will release the test questions and key answers, together which each participant answers. Participants can review their system answers and compare them with the key ones and if they think the key is wrong they can send a review claim to organizers.
Organizers will go over all the claims and decide which ones are correct and will update the test key. If finally the test data is updated, organizers will reevaluate systems against the reviewed testset and publish the final results.
Hope to count on your participation on this task that will ensure and perhaps improve the test data quality.