Task 11 Trial Data

The trial data for task 11 is stored in a spreadsheet named  task-11-trial-data.xls

This spreadsheet has two columns. The first contains the tweet-id of each trial tweet. Tweets have been selected for their figurative content, ranging from the subtly ironic to the sarcastic to the metaphorical. The second column contains a real number, between -5 … +5, that represents the sentiment of the tweet.  The actual text of each tweet is not included, due to copyright/privacy concerns that come as standard with the use of Twitter data. A script is available here for retrieving the  text of each tweet given its tweet-id:

https://github.com/aritter/twitter_download

This sentiment value is the mean sentiment assigned to the tweet by the tweet's annotators. Each tweet was annotated by seven annotators, three of which are native English speakers, the other four of which are competent non-native speakers of English. Some annotators (of both kinds) reported issues with questions of context, culture, slang, etc.

Each annotator assigned a sentiment score from the range -5 (maximal negativity, for tweets conveying obvious disgust or extreme discontent) to +5 (maximal positivity, for tweets conveying obvious joy and approval or extreme pleasure). 0 for was reserved for tweets in which the positive and negative sentiment is equally balanced. Sentiment was assigned on the basis of the intended meaning of each tweet -- the meaning the author intended the reader to unpack from the message -- and not the superficial language of the tweet. Thus, a sarcastic tweet that expressed a negative message in the language of pretend approval or delight should be marked with a negative score (as in "I just love it when my friends throw me under the bus."). Annotators was explicitly asked to consider *all* of the tweet's content when assigning a sentiment score, including any hashtags (such as #sarcasm, #irony, etc.) Participating systems will likewise be expected to use all of the tweet's content, including the hashtags.

For each tweet, the lowest annotation score and the highest annotation score was removed, to remove outlier scores and extreme disagreements between annotators. Thus, if one annotator scored a tweet as +5 and another annotated the same tweet as -5, both the +5 and -5 would be removed.  If two annotators score a tweet as +5 (or whatever the highest score for that tweet happens to be), only one of these high scores is removed. If two annotators score a tweet as -5 (or whatever the lowest score for that tweet happens to be), only one of these low scores is removed. In each case then, removing the highest and lowest annotation scores leaves 5 annotations per tweet.

The sentiment column in the trial data spreadsheet contains the mean of these "middle 5" annotations for each tweet.

The purpose of the task is for computational systems to approximate these sentiment scores as closely as possible. The sentiment column in the spreadsheet represents a target vector, and each system will be asked to estimate a vector of its own. These systems will then be judged by how "close" their vector is to the target vector, using a cosine similarity measure.