CodaLab Frequently Asked Questions
Help! My submissions are stuck in "Submitted" status!
The CodaLab team is currently working to resolve this issue: https://github.com/codalab/codalab-competitions/issues/1594. We hope it will be fully resolved by the time the official evaluation begins. The current workaround is to resubmit your results; typically it will go through after a few tries.
How many submissions am I allowed?
For the Phase 1 on the trial and/or training data, task organizers may allow as many submissions as they like.
For all official evaluation phases (i.e., phases running during the SemEval evaluation period), SemEval tasks must allow only 2 submissions per phase (i.e., set
max_submissions: 2). Since participants will immediately see the results for any system they submit, we need to restrict the submissions so that participants can’t tune to the test set by submitting a bunch of systems with different parameter settings.
How do I get my team name displayed in my leaderboard?
First, task organizers should set
allow_teams: True in their
Second, task participants should go to their "Settings", and under "Competition settings", set "Team name" to the name they are using for SemEval.
What HTML pages should task organizers prepare for CodaLab?
For SemEval-2017, we are leaving it up to task organizers which information to put on the SemEval site (http://alt.qcri.org/semeval2017/) and which information to put on the CodaLab site. As long as the information about the task, the data, and the evaluation is at one of the sites, and the two sites link to each other, that is sufficient.
If our experience with CodaLab is good this year, in future SemEvals we are likely to move more of the information to CodaLab, since that makes an easier reference point for future researchers who want to compare their results against the shared task results.
How many phases should a task declare in CodaLab?
At least 2. Phase 1 should be set up to run on the trial and/or training data and should be available as soon as you get your CodaLab site set up. This will allow your participants to test out their file formats on the trial data with the official scorer long before the evaluation period begins. Phase 2 would typically be the actual evaluation, where teams submit their results on the test data you release.
Some tasks have multiple sub-tasks, but may still need only the single Phase 2 for their evaluation phase. For example, if all of the training data was released together, and all sub-tasks are scored simultaneously by the same evaluation script, then a single Phase 2 for evaluation should be sufficient. You just need to customize your CodaLab leaderboard to show all the sub-task scores.
Tasks that plan on releasing some data during the SemEval evaluation period (09-30 Jan 2017) may want more than just Phase 1 and Phase 2. For example, if you have one evaluation setup where systems are given just raw text, and another evaluation setup where systems are given the manually annotated entities, then you could release the raw text of the test set in Phase 2, and release the manually annotated entities for the test set in a new Phase 3. We cannot extend the SemEval evaluation period, so if you have multiple phases like this, you will need to schedule all of them to fall sometime within the evaluation period.
It is also possible to have multiple competitions instead of (or in addition to) having multiple phases, but this is less common. If you think this applies to your task, please consult with the SemEval organizers.
A participant uploaded a broken submission, and it’s counting against their 2 submission limit. What should I do?
Go to the “Submissions” page of your competition, find the problematic submission, and use the “FAILED” button to mark the submission as failed. This should allow the participant to resubmit. Do not use the "DEL" button; this will delete the submission, but it will still count against their limit.
How do I withhold the results from the leaderboard until the end of the competition?
We ask that you set “leaderboard_management_mode: hide_results” so that scores will not be shared with other participants until the end of the competition. You may also set “anonymous_leaderboard: True” if, once the leaderboard results are unhidden, you do not want the participant names to show up.
Why do my participants only see a “Download your submission” link, not any of the output/error logs that would tell them if their submissions are broken?
This is because we’ve asked you to set your competition “Leaderboard Mode” to “Hide Results”. If this becomes a problem, you may change your “Leaderboard Mode” to “Default”, and then check “Anonymous leaderboard”. Participants will then see links like “View scoring output log” and “View scoring error log”. The downside is that it will also release the results (and rankings) as soon as submissions are made.
How do I let participants include more than one score on the leaderboard?
CodaLab allows only one score on the leaderboard. But as a competition organizer, you can always go to the “Submissions” page and download all the submissions (whether or not they’re on the leaderboard) using the “Download CSV” link.