Automatic Identification and Verification of Claims in Political Debates

Lab program

The presentations will be held on Thursday Sep 13 in Avignon. All hours are pm

time task authors title
3:30     Introduction to the CheckThat! lab
3:45 1 Zuo, Karakas and Banerjee A Hybrid Recognition System for Check-worthy Claims Using Heuristics and Supervised Learning
4:00 1 Hansen, Hansen, Simonsen and Lioma The Copenhagen Team Participation in the Check-Worthiness Task of the Competition of Automatic Identification and Verification of Claims in Political Debates of the CLEF-2018 Fact Checking Lab
4:15 1 Ghanem, Montes-Y-Gómez, Rangel and Rosso UPV-INAOE - Check That: Preliminary Approach for Checking Worthiness of Claims
4:30 2 Wang, Simonsen,  Larsen and Lioma The Copenhagen Team Participation in the Factuality Task of the Competition of Automatic Identification and Verification of Claims in Political Debates of the CLEF-2018 Fact Checking Lab
4:45 2 Ghanem, Montes-Y-Gómez,  Rangel and  Rosso UPV-INAOE - Check That: An Approach based on External Sources to Detect Claims Credibility

The results of the lab are available here

Overall report for both tasks.

Report on task 1

Report on task 2

The test input files for Task 1 are available here

The test input files for Task 2 are available here

The test gold labels for testing and the system submissions are available here.

The deadline to submit for task 1 has passed. No more submissions will be accepted.

The deadline to submit for task 2 has passed. No more submissions will be accepted. 

 

Important Dates

  • Registration opens: 8 November 2017
  • Registration closes: 27 April 2018
  • Beginning of evaluation cycle Task 1: 28 April 2018
  • End of evaluation cycle for Task 1: 4 May 2018
  • Beginning of evaluation cycle Task 2: 6 May 2018
  • End of evaluation cycle for Task 2: 13 May 2018
  • Results posted: 15 May 2018 (look here)TBA
  • Submission of Participant Papers [CEUR-WS]: 31 May 2018
  • Notification of Acceptance Participant Papers [CEUR-WS]: 15 June 2018
  • Camera Ready Copy of Participant Papers [CEUR-WS]: 29 June 2018
  • CEUR-WS Working Notes Preview for Checking by Authors: 18-24 July 2018
  • CLEF-2018: 10-14 September, 2018 (Avignon, France)
     

Discussion Group

Please join our discussion group clef-factcheck@googlegroups.com to receive announcements and participate in discussions. For details, questions, and discussion on a particular task, visit the task website. You will also find there a link for the task mailing list.

Rules of the Game

  1. General Rules
    1. Participation with multiple teams is not allowed.
    2. Teams are not allowed to perform manual predictions; the entire process should be fully automatic.
    3. Any effort to misuse the dataset or its source is forbidden. Further details are included in 2 below.
    4. Participants are encouraged to release the software they use (e.g., on github)
  2. Data-related rules
    1. For task 1, using external datasets with fact-checking related annotation is forbidden.  Only the released training data is allowed.
    2. For task 2, we allow any data source except from factcheck.org. For example, participants can use datasets such as
      1. Popat, et al. 2016. Credibility assessment of textual claims on the web. In Proceedings of CIKM 2016
      2. Ma, et al, 2017. Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning. In Proceedings of ACL 2017
    3. For both tasks, it is OK to use information from the Web, from Twitter, etc., but the retrieved URLs have to be checked for sanity. We include a Python script in the repository. In order for their submission to be considered valid, the participants must use this script to check any URL whose contents they would like to use. In particular, the datasets used in the following papers should not be used:
      1. Gencheva, et al. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of RANLP 2017
      2. Patwari, et al. Tathya: A Multi-Classifier System for Detecting Check-Worthy Statements in Political Debates. In Proceedings of CIKM 2017.
      3. Jaradat, et al. 2018. ClaimRank: Detecting Check-Worthy Claims in Arabic and English. In Proceedings of NAACL 2018.
  3. Evaluation rules
    1. Runs for Task 1 will be received from 28 April 0:01 until 4 May, 23:59 CET.
    2. Runs for Task 2 will be received from 5 May, 0:01 until 11 May, 23:59 CET.
    3. Participants can send multiple submissions (only the last one counts). Each one could contain one primary and up to two contrastive runs.  The participants must clearly flag each of the three. Only the last submission with the (up to) three runs will be considered as the final one.
    4. Format checkers are released as part of the dataset and will be run upon submission. A submission failing the format checking will be considered null.

@InProceedings{clef2018checkthat:overall,
    author    = {Nakov, Preslav  and  Barr\'{o}n-Cede\~{n}o, Alberto and Elsayed, Tamer and Suwaileh, Reem and M\`{a}rquez, Llu\'{i}s and Zaghouani, Wajdi and Atanasova, Pepa and Kyuchukov, Spas and Da San Martino, Giovanni},
    title     = {Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims},
    booktitle = {Proceedings of the Ninth International Conference of the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, and Interaction},
    series    = {Lecture Notes in Computer Science},
    publisher = {Springer},
    editor    = {Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian Yun Nie, Laure Soulier, Eric Sanjuan, Linda Cappellato, Nicola Ferro},
    address   = {Avignon, France},
    month     = {September},
    year      = {2018},
}

@InProceedings{clef2018checkthat:task1,
    author    = {Atanasova, Pepa and  M\`{a}rquez, Llu\'{i}s  and Barr\'{o}n-Cede\~{n}o, Alberto  and Elsayed, Tamer and Suwaileh, Reem and Zaghouani, Wajdi and Kyuchukov, Spas and Da San Martino, Giovanni and Nakov, Preslav},
    title     = {Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, Task 1: Check-Worthiness},
    booktitle = {CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum},
    series    = {{CEUR} Workshop Proceedings},
    publisher = {CEUR-WS.org},
    editor    = {Cappellato, Linda and Ferro, Nicola and Nie, Jian-Yun and Soulier, Laure},
    address   = {Avignon, France},
    month     = {September},
    year      = {2018},
}


@InProceedings{clef2018checkthat:task2,
    author    = {Barr\'{o}n-Cede\~{n}o, Alberto and Elsayed, Tamer and Suwaileh, Reem and M\`{a}rquez, Llu\'{i}s  and Atanasova, Pepa and Zaghouani, Wajdi and Kyuchukov, Spas and Da San Martino, Giovanni and Nakov, Preslav},
    title     = {Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, Task 2: Factuality},
    booktitle = {CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum},
    series    = {{CEUR} Workshop Proceedings},
    publisher = {CEUR-WS.org},
    editor    = {Cappellato, Linda and Ferro, Nicola and Nie, Jian-Yun and Soulier, Laure},
    address   = {Avignon, France},
    month     = {September},
    year      = {2018},
}

 

Published on  September 4th, 2018