SemEval-2015 Task 8

INTRODUCTION

Natural languages are filled with particular constructions for talking about spatial information, including toponyms, spatial nominals, locations that are described in relation to other locations, and movements along a path. The goals of this task include identifying and classifying items from an inventory of spatial concepts:

• Locations: regions, spatial objects, geographic and geopolitical places.
• Entities participating in spatial relations.
• Paths: routes, lines, turns, arcs.
• Topological relations: in, connected.
• Direction and Orientation: North, down.
• Time and space measurements: 20 miles away, for two hours.
• Object properties: intrinsic orientation, dimensionality.
• Frames of Reference: absolute, intrinsic, relative.
• Motion: tracking objects over time.

TASK DESCRIPTION

SpaceEval will build on the spatial role identification tasks as introduced in SemEval 2012 and used in SemEval 2013. SpacEval will consist of the following subtasks, adopting the annotation specification from ISO-Space:

1. Spatial Elements (SE):
   1. Identify spans of spatial elements including locations, paths, events and other spatial entities.
   2. Classify spatial elements according to type: PATH, PLACE, MOTION, NONMOTION_EVENT, SPATIAL_ENTITY.
   3. Identify their attributes according to type.
2. Spatial Signal Identification (SS):
   1. Identify spans of spatial signals.
   2. Identify their attributes.
3. Motion Signal Identification (MI):
   1. Identify spans of path-of-motion and manner-of-motion signals).
   2. Identify their attributes.
4. Motion Relation Identification (MoveLink):
   1. Identify relations between motion-event triggers, motion signals, and motion-event participants).
   2. Identify their attributes.
5. Spatial Configuration Identification (QSLink):
   1. Identify qualitative spatial relations between spatial signals and spatial elements.
   2. Identify their attributes.
6. Spatial Orientation Identification (OLink):
   1. Identify orientational relations between spatial signals and spatial elements.
   2. Identify their attributes.

EVALUATION

There will be three separate evaluation configurations for participants:

1. Only unannotated text is given to the user.
2. Manually annotated spatial element extents (no attributes) are given.
3. Manually annotated spatial element extents and their attributes are given.

Evaluation for each configuration is defined as follows:

1. Only Unannotated Text:
   1. SE.a: precision, recall, and F1.
   2. SE.b: precision, recall, and F1 for each type, and an overall precision, recall, and F1.
   3. SE.c: precision, recall, and F1 for each attribute, and an overall precision, recall, and F1.
   4. MoveLink.a, QSLink.a, OLink.a: precision, recall, and F1.
   5. MoveLink.b, QSLink.b, OLink.b: precision, recall, and F1 for each attribute, and an overall precision, recall, and F1.
2. Spatial Elements are Provided:
   1. SE.b and SE.c: precision, recall, and F1 for each type and its attributes, and an overall precision, recall, and F1.
   2. MoveLink.a, QSLink.a, OLink.a: precision, recall, and F1.
   3. MoveLink.b, QSLink.b, OLink.b: precision, recall, and F1 for each attribute, and an overall precision, recall, and F1.
3. Spatial Elements, their Types, and their Attributes are Provided:
   1. MoveLink.a, QSLink.a, OLink.a: precision, recall, and F1.
   2. MoveLink.b, QSLink.b, OLink.b: precision, recall, and F1 for each attribute, and an overall precision, recall, and F1.

Interested Participants

Please join the SpaceEval Google Group.