SIGLEX has currently identified 107 events (conferences and workshops) as having special interest to SIGLEX members. You can search this list by keywords or dates. You can also suggest an event to be added to this list. Proceedings of many of these workshops and conferences can be found at the ACL SIGLEX anthology. (Note that search functionality is an implied boolean AND that examines available thumbnail descriptions.)

Search terms: Year:

upcoming past all events SIGLEX events hide all


SemEval-2018: International Workshop on Semantic Evaluations

Workshop at ACL, 2018-07-01, A major NLP conference. (This workshop is endorsed by SIGLEX.) (Proceedings) (105)

SemEval-2018: International Workshop on Semantic Evaluations == Call for Task Proposals: Deadline extension == We invite proposals for tasks to be run as part of SemEval-2018. SemEval (Semantic Evaluation) is an ongoing series of evaluations of computational semantic analysis systems, organized under the umbrella of SIGLEX, the Special Interest Group on the Lexicon of the Association for Computational Linguistics. The SemEval evaluations explore the nature of meaning in natural languages in practical terms, by providing an emergent mechanism to identify the problems (e.g., how to characterize meaning and what is necessary to compute it) and to explore the strengths of possible solutions by means of standardized evaluation on shared datasets. SemEval evaluations initially focused on identifying word senses computationally, but have later grown to investigate the interrelationships among the elements in a sentence (e.g., semantic relations, semantic parsing, semantic role labeling), relations between sentences (e.g., coreference), and author attitudes (e.g., sentiment analysis), among other research directions. For SemEval-2018, we welcome any task that can test an automatic system for semantic analysis of text, be it application-dependent or application-independent. We especially welcome tasks for different languages, cross-lingual tasks, tasks requiring semantic interpretation, and tasks with both intrinsic and application-based evaluation. See the websites of previous editions of SemEval to get an idea about the range of tasks explored, e.g., for SemEval-2017: http://alt.qcri.org/semeval2017/ We strongly encourage proposals based on pilot studies that have already generated initial data which can provide concrete examples and discuss the challenges of preparing the full task. In the event of receiving many proposals, preference will be given to tasks that have already run a pilot study for the proposed task. We encourage the following aspects in task design: - Application-oriented tasks We welcome tasks that are devoted to developing novel applications of computational semantics. We will encourage tasks that have a clearly defined end-user application showcasing and enhancing our understanding of computational semantics, as well as extending the current state-of-the-art. - Umbrella tasks In order to reduce fragmentation of similar tasks and increase community effort towards solving the underlying research problems, we encourage task organizers to propose larger tasks that include several related subtasks. For example, a Semantic Similarity umbrella task might include subtasks for different kinds of similarity and different languages. Similarly, a Sentiment Analysis umbrella task might include subtasks for Twitter, Product Reviews, and Service Reviews. We also welcome task proposals for umbrella tasks focusing on different aspects of the same phenomena. For example, an Attitude Inference task might have subtasks for detecting an author’s emotional state, the sentiment of their writing, and the writing’s objectivity. In addition, the program committee will actively encourage task organizers proposing similar tasks to combine their efforts into larger umbrella tasks. Task Selection Task proposals will be reviewed by experts, and the reviews will serve as the basis for acceptance decisions. In case of conflict, more innovative new tasks will be given preference over task re-runs. Task proposals will be evaluated on: - Interest: Is the proposed task likely to attract a sufficient number of participants? - Data: Are the plans for collecting data convincing? Will the resulting data be high quality? Will the data annotation be ready on time? - Evaluation: Is the methodology for evaluation sound? Is the necessary infrastructure available or can it be built in time for the shared task? - Impact: What is the expected impact of the data in this task on future research beyond the SemEval Workshop? Task Organization Task organizers are expected to provide to task participants format checkers and standard scorers. Moreover, in order to lower the obstacles to participation, we encourage task organizers to provide baseline systems that participants can use as a starting point. A baseline system typically contains code that reads the data, creates a baseline response (e.g., random guessing, majority class prediction, etc.), and outputs the evaluation results. Whenever possible, baseline systems should be written in widely used programming languages and/or should be implemented as a component for standard NLP pipelines such as UIMA or GATE. New Tasks vs. Task Reruns We welcome both new tasks and task reruns. For a new task, a major concern to be addressed in the proposal is whether it would be able to attract participants. For task reruns, the organizers should in their proposal defend the need for another iteration of their task, explain, for example, why there is need for a new form of evaluation (e.g., a new metric to test new phenomena, a new application-oriented scenario, etc.) or need to test on new types of data (e.g., social media, domain-specific corpora), whether there is significant expansion in scale over a previous trial run of the task, etc. In the case of a rerun, we further discourage carrying over the same tasks year after year and just adding new subtasks as this can lead to the accumulation of too many subtasks. Evaluating on a different dataset with the same task formulation typically should not be considered a separate subtask. Tasks that have already run for three years will not be accepted for SemEval-2018. If however the organizers estimate there is need for another iteration of their task, they are welcome to submit a task rerun proposal for SemEval-2019 (the calendar for submissions will be announced in Feb, 2018). Solid justification for the re-run will be needed highlighting its novel aspects compared to previous editions, in respect to the criteria discussed above. NEW DATES for SemEval-2018 Task proposals due April 21, 2017 Task selection notification June 5, 2017 Tasks merged June 25, 2017 Trial data ready August 14, 2017 Training data ready September 18, 2017 Test data ready December 1, 2017 Evaluation start January 10, 2018 Evaluation end January 31, 2018 Paper submission due February 28, 2018 Paper reviews due March 31, 2018 Camera ready due April 30, 2018 SemEval workshop Summer 2018 The SemEval-2018 Workshop will be co-located with a major NLP conference in 2018. Tasks that fail to keep up with crucial deadlines such as the dates for having the task and CodaLab website up and dates for uploading trial, training, and test data may be cancelled at the discretion of SemEval organizers. While consideration will be given to extenuating circumstances, our goal is to provide sufficient time for the participants to develop strong and well-thought-out systems. Cancelled tasks will be encouraged to submit proposals for the subsequent year’s SemEval. Submission Details The task proposals should be a self-contained document of roughly 4-8 pages. Each proposal should contain the following: - Overview -- A summary of the task in general -- Motivation, why this task is needed and which communities would be interested in participating -- What the expected impact of the task will be - Data & Resources -- How the training/testing data will be built and/or procured -- What source texts/corpora are going to be used? Please discuss whether existing corpora have been re-used or not. -- How much data is going to be produced -- How will quality of the data be ensured and evaluated -- An example of how a data instance would look like -- The anticipated availability of the necessary resources to the participants (copyright, etc.) -- The resources required to prepare the task (computation and annotation time, costs of annotations, etc.) and their availability - Pilot Task -- Details of the pilot task, if any -- What lessons were learned and how these will impact the future task design - Evaluation -- The evaluation methodology to be used, including clear evaluation criteria - For Task Reruns -- Justification for why a new iteration of the task is needed, using the criteria discussed above -- What will differ from the previous instance -- The expected impact of the re-run compared with the previous instance - Task organizers -- Names, affiliations, brief description of research interests and relevant experience, contact information (email). Proposals will be reviewed by an independent group of area experts who may not have familiarity with recent SemEval tasks and therefore, all proposals should be written in a self-explanatory manner and contain sufficient examples. Submission will be electronic in PDF format through the START conference management system at: https://www.softconf.com/acl2017/semeval/ Please use the SemEval 2018 Task Proposal Submission page. In case you are not sure whether a task is suitable for SemEval, please feel free to get in touch with the SemEval organizers at semeval- organizers@googlegroups.com to discuss your idea. Chairs: Marianna Apidianaki, LIMSI-CNRS & University of Pennsylvania Saif M. Mohammad, National Research Council Canada Steven Bethard, University of Alabama at Birmingham Marine Carpuat, University of Maryland The SemEval discussion group: Please join our discussion group at semeval3@googlegroups.com in order to receive announcements and participate in discussions. The SemEval-2018 Website: http://alt.qcri.org/semeval2018/ Contact: semeval-organizers@googlegroups.com