Data and Tools


For Tasks 1 and 2 we shall use the ShARe corpus containing clinical notes from MIMIC II database manually annotated for disorder mentions and normalized to an UMLS Concept Unique Identifier (CUI) when possible.
The corpus annotation guidelines contain more details and examples.


Data Format

The annotations follow a pipe-delimited stand-off character-offset format. The template will contain the following format:


report name|disorder-span|cui|Norm_NI|Cue_NI|Norm_SC|Cue_SC|Norm_UI|Cue_UI|

System results should be submitted in the same format as shown below. In addition, in order to simplify scoring, we will ask participants to format the submissions in a particular directory structure so that it would help facilitate automating the scoring process. It is harder to score multiple systems automatically when they follow different formats. We will make this structure available to you during the evaluation period.  We will also provide a validation script that would identify common errors that we had noticed in the previous scoring runs, and which will make the scoring process smoother.


*NULL|*false|*NULL| *unmarked|*NULL|severe|*NULL|*false|*NULL|C0040132|*NULL|

Trial Data

The following tarball contains four documents along with their annotations:



Training Data

Clinical data -- even in it's de-identified form -- has various privacy controls in place.  In order to get the annotation along with the associated clinical notes, a participant would have to follow the following steps to ensure that they understand the ethical aspects of handling human subjects data. This training is free.

  1. Register to participate.This form is separate from the main SemEval registration form.
  2. Obtain a human subjects training certificate.
          If you do not have a certificate, you can take the CITI training course or the NIH training course
  3. Go to the Physionet website
  4. Click on the link for “creating a PhysioNetWorks account” (near middle of page)
          and follow the instructions.
  5. Once you have a login and password, go to MIMIC II and accept the terms of a Data
          Use Agreement (DUA)
  6. You will receive an email telling you to fill in your information on the DUA and email
          it back with your human subjects training certificate.
  7. Fill out the DUA using the word “SemEval-2015” in the description of the project
          and mail it back (pasted into the email) with your human subjects certificate attached.
  8. Once you are approved to use MIMIC II, you can then go to your PhysioNetWorks Home
          as shown below

  9. Once here, you will see a list of all projects.
  10. Select the link to "SemEval 2015 -- Analysis of Clinical Text"
  11. Apply for access to the data by clicking on the link "here" shown below

  12. Once you do this, the organizers will get a request to add you to the project.
  13. After the organizers give you access, you will get an email informing you that you can
           access the data.

UMLS Knowledge Source

In case you haven't already have access to the UMLS knowledge sources, you would have to register at the following NIH website and request a license:


Once you have obtained the license you can download the UMLS release files from the following URL:


Here you will find multiple different versions. NIH updates these about two times in a year. The version of the database used for the ShARe annotations is 2012AB.  This resource contains mapping for many different terminologies.  The ones that are relevant in our case are the UMLS CUIs for the SNOMED-CT terminology.

Contact Info

Organizers (in alphabetical order)

  • Wendy W. Chapman, University of Utah
  • Noemie Elhadad, Columbia University
  • Suresh Manandhar, University of York, UK
  • Sameer S. Pradhan, Harvard University
  • Guergana K. Savova, Harvard University

  • Contact:
  • Other Info