Results

  • English STS results

Team Name deft-forum deft-news headlines images OnWN tweet-news Weighted mean Rank
Bielefeld_SC-run1 0.2109 0.4315 0.3208 0.3676 0.3667 0.4146 0.3538 35
Bielefeld_SC-run2 0.2108 0.4307 0.3112 0.3558 0.3607 0.4087 0.3470 36
BUAP-EN-run1 0.4557 0.6855 0.6888 0.6966 0.6539 0.7706 0.6715 19
DLS@CU-run1 0.4828 0.7657 0.7646 0.8214 0.7227 0.7639 0.7337 7
DLS@CU-run2 0.4828 0.7657 0.7646 0.8214 0.8589 0.7639 0.7610 1
FBK-TR-run1 0.3219 0.5231 0.5469 0.6009 0.6615 0.4625 0.5348 28
FBK-TR-run2 0.1670 0.4214 0.4854 0.5211 0.5725 0.3588 0.4413 31
FBK-TR-run3 0.3051 0.4046 0.4712 0.4891 0.5512 0.4378 0.4588 30
IBM_EG-run1 0.4742 0.7431 0.7371 0.8012 0.7603 0.7296 0.7220 8
IBM_EG-run2 0.4645 0.6412 0.7102 0.7471 0.7322 0.6960 0.6841 15
LIPN-run1 0.4544 0.6402 0.6527 0.8094 - 0.5507 0.5083 29
LIPN-run2 0.0843 - - - - - 0.0101 38
Meerkat_Mafia-Hulk 0.4495 0.7850 0.7571 0.7896 0.7872 0.7571 0.7349 6
Meerkat_Mafia-pairingWords 0.4711 0.7628 0.7597 0.8013 0.8745 0.7793 0.7605 2
Meerkat_Mafia-SuperSaiyan 0.4918 0.7712 0.7666 0.7676 0.8022 0.7651 0.7410 5
NTNU-run1 0.4369 0.7138 0.7219 0.8000 0.8348 0.4109 0.6631 21
NTNU-run2 0.5084 0.7656 0.7525 0.8129 0.7767 0.7921 0.7491 4
NTNU-run3 0.5305 0.7813 0.7837 0.8343 0.8502 0.6755 0.7549 3
RTM-DCU-run1* 0.4341 0.6974 0.6199 0.6995 0.8058 0.6882 0.6706 20
RTM-DCU-run2* 0.3965 0.6811 0.6125 0.6656 0.7992 0.6691 0.6513 23
RTM-DCU-run3* 0.3078 0.5562 0.6301 0.6475 0.8004 0.5531 0.6076 27
SemantiKLUE-run1 0.3373 0.6077 0.7283 0.7833 0.8482 0.6319 0.6874 14
SemantiKLUE-run2 0.3486 0.6429 0.7332 0.7728 0.8550 0.6403 0.6935 13
StanfordNLP-run1 0.3186 0.6347 0.6361 0.7583 0.6269 0.6685 0.6270 24
StanfordNLP-run2 0.3035 0.6791 0.6208 0.7149 0.6250 0.6362 0.6101 26
StanfordNLP-run3 0.3423 0.6503 0.6021 0.7540 0.6087 0.6380 0.6137 25
UMCC_DLSI_SemSim-run1 0.4752 0.6619 0.6318 0.7421 0.8127 0.6753 0.6823 16
UMCC_DLSI_SemSim-run2 0.4689 0.6622 0.6255 0.7390 0.8140 0.6536 0.6756 18
UMCC_DLSI_SemSim-run3 0.2826 0.3854 0.2669 0.4359 0.6028 0.2780 0.3815 33
UNAL-NLP-run1 0.5043 0.7205 0.7616 0.8071 0.7823 0.6145 0.7113 12
UNAL-NLP-run2 0.3826 0.7305 0.7645 0.7706 0.8268 0.4028 0.6573 22
UNAL-NLP-run3 0.4607 0.7216 0.7605 0.7782 0.8426 0.6583 0.7209 9
UNED-run22_p_np 0.1043 0.3148 0.0374 0.3243 0.5086 0.4898 0.3097 37
UNED-runS5K_10_np 0.1181 0.5059 0.0570 0.4981 0.4880 0.5794 0.3791 34
UNED-runS5K_3_np 0.0941 0.5644 0.0177 0.6070 0.5765 0.6700 0.4307 32
UoW-run1 0.3419 0.7512 0.7535 0.7763 0.7990 0.7368 0.7143 11
UoW-run2 0.3419 0.5875 0.7535 0.7877 0.7990 0.6281 0.6817 17
UoW-run3 0.3419 0.7634 0.7535 0.7877 0.7990 0.7529 0.7207 10

 

One team submitted non-uniform confidence scores. The following table shows the results and rankings when using the submitted confidence scores:

 

Team Name deft-forum deft-news headlines images OnWN tweet-news Weighted mean Rank
RTM-DCU-run1* 0.4181 0.6846 0.6216 0.6981 0.8331 0.6870 0.6729 19
RTM-DCU-run2* 0.3831 0.6739 0.6094 0.6629 0.8260 0.6691 0.6534 23
RTM-DCU-run3* 0.2731 0.5526 0.6330 0.6441 0.8246 0.5683 0.6110 26

Notes: 

- : Not submitted. 

* : Post-deadline submission.

  • Spanish STS results

 

Team Name System type Wikipedia News Weighted correlation Rank
Bielefeld_SC-run1 unsupervised* 0.2632 0.55445 0.43708 22
Bielefeld_SC-run2 unsupervised* 0.26458 0.55455 0.4377 21
BUAP-run1 supervised 0.5504 0.6785 0.62688 17
BUAP-run2 unsupervised 0.63964 0.76369 0.7137 14
RTM-DCU-run1 supervised 0.42164 0.70003 0.58784 18
RTM-DCU-run2 supervised 0.36886 0.62527 0.52194 20
RTM-DCU-run3 supervised 0.42424 0.64113 0.55373 19
LIPN-run1 supervised 0.65194 0.82554 0.75558 11
LIPN-run2 supervised 0.71647 0.8316 0.7852 6
LIPN-run3 supervised 0.71618 0.80857 0.77134 10
Meerkat_Mafia-run1 unsupervised 0.6682 0.78517 0.73803 13
Meerkat_Mafia-run2 unsupervised 0.74305 0.84542 0.80417 2
Meerkat_Mafia-run3 supervised 0.73815 0.82247 0.78849 5
TeamZ-run1 supervised 0.6102 0.71654 0.67369 15
TeamZ-run2 supervised 0.60425 0.70974 0.66723 16
UMCC_DLSI-run1 supervised 0.74084 0.82539 0.79132 4
UMCC_DLSI-run2 supervised 0.78021 0.82539 0.80718 1
UNAL-NLP-run1 weakly supervised 0.78036 0.81541 0.80129 3
UNAL-NLP-run2 supervised 0.75659 0.78293 0.77232 9
UNAL-NLP-run3 supervised 0.68936 0.79648 0.75331 12
UoW-run1 supervised 0.74826 0.80008 0.7792 7
UoW-run2 supervised 0.74826 0.80008 0.7792 8

Notes: 
1. The main evaluation column is "mean". The rank column gives the rank of the submission as ordered by the "mean" result.
2. * denotes a system that used Wikipedia to build its model for the Wikipedia test dataset.

 

 

 

 

 

 

 

 

Contact Info

  • STS-En: admin-sts-en{at}googlegroups{dot}com
  • STS-Es: admin-sts-es{at}googlegroups{dot}com

Other Info

Announcements

Copyright 2024 - SemEval-2014 Task 10. All Right Reserved

Designed by BhMad Studio  |  Powered by GetSimple