Papers
Both task organizers and participants are to submit papers describing the tasks and their systems respectively. These will be part of the official SemEval proceedings.
Length of papers:
Task organizers are to submit an 8-page task-description paper describing their task, data, evaluation, results, and a summary of participating systems. In case of extenuating circumstances such as a large number of sub-tasks, more than 8-pages is acceptable.
Participants are to submit a 4-page system description paper describing their system and submission(s). If you are participating in multiple SemEval tasks or in multiple sub-tasks within a task, then 6-pages is acceptable. For example, if two similar systems produced by the same team are used to make submissions to Task X and Task Y, then they should write a single system-description paper and its length can exceed four pages (in this case, they can aim for six pages). If the same team, or two teams with overlapping team members, create two very different systems for two very different tasks, then they should write two separate papers (one for each task; and each four pages long).
References do not count against the page limits (8 pages for task descriptions, 4 pages for system descriptions). You may have as many additional pages of references as you want.
Due Dates:
System Description Papers (written by SemEval participants) are due Mon 05 Mar 2018 by 23:59 GMT -12:00.
Task Description Papers (written by SemEval task organizers) are due Mon 12 Mar 2018 by 23:59 GMT -12:00.
See detailed schedule of reviewing deadline, acceptance notification date, and camera-ready deadline on the home page (Important Dates).
Submission Website:
Submission is electronic, using the Softconf START conference management system at: https://www.softconf.com/naacl2018/SemEval-2018/
Style Files:
Since SemEval is at NAACL this year, we will be using the NAACL-2018 style templates with two important modifications:
- The review process is single-blind. Submissions are not anonymous and should use the ACL camera-ready formatting.
- Paper titles are required to follow the SemEval template: for system description papers submitted by task participants: "[SystemName] at SemEval-2018 Task [TaskNumber]: [Insert Paper Title Here]", for task description papers submitted by task organizers: ``SemEval-2018 Task [TaskNumber]: [Task Name]''.
Latex templates adapted for SemEval can be found here
The original NAACL templates (including MS Word instructions) can be found at http://naacl2018.org/call_for_paper.html.
Non-Anonymous Reviewing
Reviewing of papers is not anonymous. You are to enter your names and affiliations on the paper (be it the system-description paper or the task-description paper).
Access to the Task-Description Paper
We will set things up such that all participants of a task can view the task description paper when it is uploaded for review.
FAQ:
Q. What to include in a system-description paper?
A. Here are some key pointers:
- Replicability: present all details that will allow someone else to replicate your system
- Analysis: focus more on results and analysis and less on discussing rankings; report results on several runs of the system (even beyond the official submissions); present ablation experiments showing usefulness of different features and techniques; show comparisons with baselines.
- Duplication: cite the task description paper; you can avoid repeating details of the task and data, however, briefly outlining the task and relevant aspects of the data is a good idea. (The official BibTeX citations for papers will not be released until the camera-ready submission period, so during the initial submission, please use some sort of placeholder citation.)
It may be helpful to look at some of the papers from past SemEvals, e.g., from https://aclweb.org/anthology/S/S16/ and https://aclweb.org/anthology/S/S17/.
Q. What to focus on in a task-description paper?
A. Here are some key pointers:
- Replicability: present all details that will allow someone else to replicate the data creation process and evaluation.
- Analysis: focus more on results and analysis and less on discussing rankings
- Summary of systems: Summarize the techniques, features, and resources used. Highlight what tended to work and what did not, across the systems.
It may be helpful to look at some of the papers from past SemEvals, e.g., from https://aclweb.org/anthology/S/S16/ and https://aclweb.org/anthology/S/S17/.
Q. Can we change the name of our team after the evaluation results are announced?
A. Contact your task organizer for this. This is usually not a problem, and has been allowed in the past.