Frequently Asked Questions


Q: I have questions pertaining to SemEval and its tasks. How do I find answers?
A: For questions specific to a task: browse the task webpage, join the task mailing list, look if the question has already been responded to in the mailing list. If you still do not find the answer to your question, post it on the task mailing list.
For general SemEval questions: browse the SemEval webpages, including this FAQ page. If you still do not find the answer to your question, email semeval-organizers@googlegroups.com.

Q: How many runs can be sent for each task?
A: Task organizers often specify how many submissions by each team will be evaluated. Email your task organizers if these details are not already stated on the task webpage.

Q: Can I take part in a Semeval Task as an individual?
A: Yes.

Q: Can I make submissions to multiple SemEval tasks?
A: Yes, the same person or team can make submissions to multiple SemEval tasks. However, individual task organizers might place restrictions on how many teams the same person can be involved in for their task. For example, they might stipulate that one cannot make submissions as part of more than one team for their task. That is, they may state that a person cannot be in two separate teams that both make submissions to the same task. Task organizers often specify how many submissions by each team will be evaluated, and the above stipulation is a way to avoid abusing that by simply creating multiple teams. Email your task organizers if these details are not already clearly stated on the task webpage.

Q: What is the evaluation period for the task I am interested in?
A: 8 Jan to 29 Jan 2017 is the period during which the task organizers must schedule the evaluation periods for their individual tasks. Usually, evaluation periods for individual tasks is 7 to 14 days, but there is no hard and fast rule about this. Contact the task organizers for the tasks you are interested in for the exact time frame when they will conduct their evaluations. They should tell you the date by which they will release the test data, and the date by which participant submissions are to be uploaded. Note that some tasks may involve more than one sub-tasks, each having a separate evaluation time frame.

Q: Do I have to submit a paper to the SemEval workshop? Do I have to register for the SemEval workshop?
A: While we strongly encourage all participants to attend the workshop, it is not required to either register or attend the workshop for your system paper to be included in the proceedings.

Q: Can I submit system output after the competition deadline?
A: You could still request the task organizers to evaluate your system output, however, agreeing to do so is up to them. If the deadline has passed, the best option is to use the evaluation script and gold answers available on the task website to evaluate your system output yourself.

Q: Is participation in the tasks and workshop free?
A: Submitting a system output to SemEval tasks is free.  You can attend the workshop by registering for the workshop, and it comes with a registration fee.

Contact Info

Organizers

  • Marianna Apidianaki, LIMSI, CNRS, University Paris-Saclay & University of Pennsylvania
  • Saif M. Mohammad, National Research Council Canada
  • Jonathan May, ISI, University of Southern California
  • Ekaterina Shutova, University of Cambridge
  • Steven Bethard, University of Alabama at Birmingham
  • Marine Carpuat, University of Maryland

Email

semeval-organizers@googlegroups.com Note that this is the mailing list for SemEval organizers. For questions on a particular task, post them at the *task* mailing list or contact the task organizers directly. You can find the task mailing list from the task webpage.

Other Info

Announcements

  • Apr. 22: A draft of the program is now available
  • Mar. 15: On-site childcare is available throughout all of SemEval! Register here
  • Feb. 23: Deadlines for paper and review submission have changed. Please see the main page or look for email from your task organizer (or, for task organizers, from the semeval organizers)
  • Nov. 20: SemEval-2018 will be co-located with NAACL HLT 2018 in New Orleans. The workshop will be on June 5-6, 2018.