Papers

 

Both task organizers and participants are to submit papers describing the tasks and their systems respectively. These will be part of the official SemEval proceedings.

 

Length of papers
Task organizers are to submit an 8-page task-description paper describing their task, data, evaluation, results, and a summary of participating systems. In case of extenuating circumstances such as a large number of sub-tasks, more than 8-pages is acceptable.

Participants are to submit a 4-page system description paper describing their system and submission(s). If you are participating in multiple SemEval tasks or in multiple sub-tasks within a task, then 6-pages is acceptable. For example, if two similar systems produced by the same team are used to make submissions to Task X and Task Y, then they should write a single system-description paper and its length can exceed four pages (in this case, they can aim for six pages). If the same team, or two teams with overlapping team members, create two very different systems for two very different tasks, then they should write two separate papers (one for each task; and each four pages long).

References do not count against the page limits (8 pages for task descriptions, 4 pages for system descriptions). You may have as many additional pages of references as you want.

 

Important Style Requirements

To prepare your camera-ready version:

  • Since SemEval is at NAACL this year, we will be using the NAACL-2019 style templates with some important modifications for SemEval stated below. The style file of NAACL-2019 can be found here:

            https://naacl2019.org/calls/papers/

  • Specifically for SemEval, paper titles are required to follow the SemEval template: for system description papers submitted by task participants: "[SystemName] at SemEval-2019 Task [TaskNumber]: [Insert Paper Title Here]",  for task description papers submitted by task organizers: ``SemEval-2019 Task [TaskNumber]:  [Task Name]''.

To prepare your submission for review (this phase has already passed):

  • The review process is single-blind. Submissions for review are not anonymous and should use the NAACL camera-ready formatting. Latex templates adapted for SemEval can be found at: http://alt.qcri.org/semeval2018/data/uploads/semeval18-latex.tgz

 

Due Dates

See detailed schedule of reviewing deadline, acceptance notification date, and camera-ready deadline on the home page (Important Dates).


Submission Website

https://www.softconf.com/naacl2019/SemEval

 

Non-Anonymous Reviewing

Reviewing of papers is not anonymous. You are to enter your names and affiliations on the paper (be it the system-description paper or the task-description paper).

 

Access to the Task-Description Paper
We will set things up such that all participants of a task can view the task description paper when it is uploaded for review. 


FAQ:

Q. What to include in a system-description paper?
A. Here are some key pointers:

  • Replicability: present all details that will allow someone else to replicate your system
  • Analysis: focus more on results and analysis and less on discussing rankings; report results on several runs of the system (even beyond the official submissions); present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
  • Duplication: cite the task description paper; you can avoid repeating details of the task and data, however, briefly outlining the task and relevant aspects of the data is a good idea. (The official BibTeX citations for papers will not be released until the camera-ready submission period, so during the initial submission, please use some sort of placeholder citation.)

It may be helpful to look at some of the papers from past SemEvals, e.g., from https://aclweb.org/anthology/S/S16/ and https://aclweb.org/anthology/S/S17/.

Q. What to focus on in a task-description paper?
A. Here are some key pointers:

  • Replicability: present all details that will allow someone else to replicate the data creation process and evaluation.
  • Analysis: focus more on results and analysis and less on discussing rankings
  • Summary of systems: Summarize the techniques, features, and resources used. Highlight what tended to work and what did not, across the systems.

It may be helpful to look at some of the papers from past SemEvals, e.g., from https://aclweb.org/anthology/S/S16/ and https://aclweb.org/anthology/S/S17/.

Q. Can we change the name of our team after the evaluation results are announced?
A. Contact your task organizer for this. This is usually not a problem, and has been allowed in the past.

Contact Info

Organizers

  • Jonathan May, ISI, University of Southern California
  • Ekaterina Shutova, University of Amsterdam
  • Aurelie Herbelot, University of Trento
  • Xiaodan Zhu, Queen's University
  • Marianna Apidianaki, LIMSI, CNRS, Université Paris-Saclay & University of Pennsylvania
  • Saif M. Mohammad, National Research Council Canada

Email

semeval-organizers@googlegroups.com Note that this is the mailing list for SemEval organizers. For questions on a particular task, post them at the *task* mailing list or contact the task organizers directly. You can find the task mailing list from the task webpage.

Other Info

Announcements

  • 2019/5/1: Our workshop program and proceedings are available now. Please click here for program, and click here for proceedings. They are also available under the "Workshop" menu.
  • 2019/2/17: We are excited to announce that Sam Bowman from NYU will give a keynote at the workshop. Please click here for details.
  • 2019/2/6: For visa application, you can visit NAACL-2019 Participants Webpage. At the bottom of the page, you can find the visa letter request form.
  • 2018/12/6: SemEval-2019 will be held June 6-7, 2019 in Minneapolis, USA, collocated with NAACL-HLT 2019.
  • 2018/9/4: Microsoft is co-sponsoring SemEval! More