Data and Tools


CORPORA

TExEval-2 is not restricted to any specific corpora, but participants are encouraged to use Wikipedia documents that are available as XML dumps for English, Dutch, French and Italian. The Wikipedia Extractor python script can be used to clean the text using the following Linux instruction:


bzcat enwiki-latest-pages-articles.xml.bz2 |  python WikiExtractor.py -cb 250K -o extracted


TEST DATA

An archive with the complete test data including gold standard taxonomies can be downloaded here.


Test data is released for the following subtasks:

  • Taxonomy construction
  • Hypernym identification
  • Multilingual taxonomy construction
  • Multilingual hypernym identification

TExEval-2_testdata_EN_1.0

Download the English test terms here. The trial data package contains the following:
README.txt                                textual description of this package
environment_eurovoc_en.terms    terms from the EuroVoc taxonomy rooted on "environment"
food_en.terms                            terms from a manual taxonomy rooted on "food"
food_wordnet_en.terms               terms from the WordNet taxonomy rooted on "food"
science_en.terms                        terms from a manual taxonomy rooted on "science"
science_eurovoc_en.terms            terms from the EuroVoc taxonomy rooted on "science"
science_wordnet_en.terms            terms from the WordNet taxonomy rooted on "science"


TExEval-2_testdata_NL_1.0

Download the Dutch test terms here.


TExEval-2_testdata_FR_1.0

Download the French test terms here.


TExEval-2_testdata_IT_1.0

Download the Italian test terms here.

 

TRIAL DATA

TExEval_trialdata_1.2

Download the trial data here.

The trial data package contains the following:

README.txt                      A description
ontolearn_AI.taxo            Artificial Intelligence taxonomy [¹]
ontolearn_AI.taxo.eval   Human evaluation for the Artificial Intelligence taxonomy relations [¹]
WN_plants.taxo               WordNet plants taxonomy
WN_plants.terms             WordNet plants terminology
WN_vehicles.taxo           WordNet vehicles taxonomy
WN_vehicles.terms         WordNet vehicles terminology

 

Download the trial Dutch terms and taxonomy:
WN_vehicles_dutch.taxo           Dutch WordNet vehicles taxonomy
WN_vehicles_dutch.terms         Dutch WordNet vehicles terminology

 

Download the trial French terms and taxonomy:

WN_vehicles_french.taxo           French WordNet vehicles taxonomy
WN_vehicles_french.terms         French WordNet vehicles terminology

 

Download the trial Italian terms and taxonomy:

WN_vehicles_italian.taxo           Italian WordNet vehicles taxonomy
WN_vehicles_italian.terms         Italian WordNet vehicles terminology

 

** Special characters might not be displayed correctly if you view the files in your browser.

 

FILE FORMAT

The input file format for taxonomies (.taxo) is tab-separated fields:

relation_id <TAB> term <TAB> hypernym

where:
- relation_id: is a relation identifier;
- term: is a term of the taxonomy;
- hypernym: is a hypernym for the term.

e.g

0<TAB>cat<TAB>animal
1<TAB>dog<TAB>animal
2<TAB>car<TAB>animal
....

The input files format for the system relation evaluation (.taxo.eval) is tab-separated fields:

relation_id <TAB> eval

where:
- relation_id: is a relation identifier;
- eval: is an empty string if the relation is good, an "x" otherwise

e.g.
0<TAB>
1<TAB>
2<TAB>x
....


The input file format for terminologies (.terms) is tab-separated fields:

term_id <TAB> term

where:
- term_id: is a term identifier;
- term: is a domain term.

 

TExEval_tool_1.0

Download the tool package here.

The tool package contains the following files:

README.txt                      A description file
TExEval.jar                        Program for scoring the outputs
runExample.sh                   Linux script for running the example evaluation
example/gold1.taxo            Example gold standard taxonomy
example/sys1.taxo             Example of a taxonomy produced by a system
example/sys1.taxo.eval     Example file that includes manually evaluated relations from the system taxonomy
example/results.txt             Example output of the scoring system

 

INPUT FORMAT

The input file format for the system and gold standard taxonomies is tab-separated fields:

relation_id <TAB> term <TAB> hypernym

where:
- relation_id: is a relation identifier;
- term: is a term of the taxonomy;
- hypernym: is a hypernym for the term.

e.g

0<TAB>cat<TAB>animal
1<TAB>dog<TAB>animal
2<TAB>car<TAB>animal
....

The input files format for the system relation evaluation is tab-separated fields:

relation_id <TAB> eval

where:
- relation_id: is a relation identifier;
- eval: is an empty string if the relation is good, an "x" otherwise

e.g.
0<TAB>
1<TAB>
2<TAB>x
....

 

EVALUATION METRICS

The TExEval.jar is a runnable jar, which evaluates a system generated taxonomy against a gold standard taxonomy. The measures reported by the program are:
1) A measure to compare the overall structure of the taxonomy against a gold standard, with an approach used for comparing hierarchical clusters[¹];
2) Precision: the number of correct relations over the number of given relations;
3) Recall: the number of relations in common with the gold standard over the number of gold standard relations;


To run TExEval.jar on your linux machine, open a terminal and enter:
"java -jar TExEval.jar system.taxo goldstandard.taxo root results"
or
"java -jar TExEval.jar system.taxo.eval results"

where:
- system.taxo: is the taxonomy produced by your system;
- system.taxo.eval: is the evaluation of the system produced relations;
- goldstandard.taxo: is the gold standard taxonomy;
- root: is the common root node for the system and the goldstandard taxonomies
- result: is the file where the program will write the results.

By running the runExampleVSGoldStandard.sh, the TExEval.jar compare the following system produced taxonomy:

example/sys1.taxo
0 a entity
1 b a
2 c b
3 d b
4 e b

against the following gold standard taxonomy:

example/gold1.taxo
0 a entity
1 b a
2 c b
3 d b
4 e b
5 f e
6 g e
7 h e

producing the following result.txt file

example/results.txt
Taxonomy file ./example/sys1.taxo
Gold Standard file ./example/gold1.taxo
Root entity
level B Weight BxWeight
0 0.18257418583505536 1.0 0.18257418583505536
1 0.18257418583505536 0.5 0.09128709291752768
2 0.18257418583505536 0.3333333333333333 0.06085806194501845
3 0.0 0.25 0.0
Cumulative Measure 0.16066528353484874
Recall from relation overlap 0.625

where:
1) the two first lines report the arguments passed to the jar application
2) a structural comparison of the system taxonomy against the gold standard taxonomy[¹]
3) the estimated Recall


By running the runExamplePrecision.sh, the TExEval.jar compute the Precision from the following Evaluation file for the system produced relation:

example/sys1.taxo.eval
0
1
2
3
4 x

and produce the following result.txt file:

Taxonomy relation evaluation file ./example/sys1.taxo.eval
Precision from relation evaluation 0.8

 

TAXONOMY VISUALISATION

To visualise a taxonomy as a graph we recommend to convert the taxonomy file to the .dot format and to use the Gephi visualization and exploration platform. This tutorial should give you the basics. For example try using the Force Atlas 2 algorithm to layout the graph and node degree for node size and colour. Check the "Prevent Overlap" option and display node labels using the "Node size" option for label size. Check this paper [²] for more information about visualising taxonomies. Below you can find the result for the vehicles taxonomy provided as trial data.

[¹] Paola Velardi, Stefano Faralli, Roberto Navigli. OntoLearn Reloaded: A Graph-based Algorithm for Taxonomy Induction. Computational Linguistics, 39(3), MIT Press, 2013, pp. 665-707.

[²] Hooper, Clare J., Georgeta Bordea, and Paul Buitelaar. "Web science and the two (hundred) cultures: representation of disciplines publishing in web science." In Proceedings of the 5th Annual ACM Web Science Conference, pp. 162-171. ACM, 2013.

Contact Info

Organizers

  • Georgeta Bordea - Insight, Centre for Data Analytics, National University of Ireland, Galway
  • Els Lefever - LT3 language and translation team at the Faculty of Arts and Philosophy at Ghent University
  • Paul Buitelaar - Insight, Centre for Data Analytics, National University of Ireland, Galway

email :

Other Info

Announcements

  • The deadline for submitting system description papers was extended to March 4, 2016
  • Final rankings made available on February 5, 2016
  • English, Dutch, French and Italian test data released on December 15, 2015
  • The task is organised in an unsupervised setting therefore no training data will be provided
  • Updated Dutch trial taxonomy and released Italian and French trial terms and taxonomy on July 31, 2015
  • Dutch trial terms and taxonomy released on July 12, 2015
  • Trial data and tools released on June 30, 2015