Representing Multilevel Sentiment Annotated Data

Introduction:

We present a newly collected training dataset of 8000 figurative English tweets. The dataset consists of ironic, sarcastic and metaphorical tweets. The corpus is annotated manually on an 11 point scale (-5 to +5, including 0). This brief report highlights some initial findings on the annotated data which is intended to serve as training data for a sentiment analysis task on figurative tweets. The data was annotated using the CrowdFlower crowd-sourcing service, which has many advantages and some disadvantages.

Note: We are releasing the training data in two formats:  integer sentiment scores on the 11 point scale  (-5 to +5, including 0), and continuous real-valued scores on the same scale.

Dataset:

We have used the Twitter4j api to collect the dataset, which allows one to harvest a stream of real-time tweets by using queries to search over tweet content. In order to collate a suitable corpus for a figurative Sentiment Analysis task, we have prepared a set of search queries and a list of hashtags to retrieve the tweets. 
The tweets were collected over a time period of 4 weeks from 1st Jun, 2014 to 30th Jun, 2014. Tweets contain information in different media such as images. To clean such information, a post-processing method is used which eliminates tweets containing URLs, images, emails etc. 

Very short tweets were eliminated using the following criteria: the minimum length of each tweet is 30 characters excluding Hashtags, and 40 characters including Hashtags. 

The final dataset consists of 8000 tweets collected from many different Twitter users. The dataset contains 5000 sarcastic tweets, 1000 ironical tweets and 2000 metaphorical tweets.

Corpus Annotation using CrowdFlower:

The 8000 tweets were annotated by using a crowd-sourcing web service, named CrowdFlower. The biggest problem with using such services is scammers -- users who do not properly engage with the task at hand. To prevent or thwart scammers, a initial gold standard corpus was prepared consisting of 1200 tweets, annotated by 7 researchers using an 11 points scale (-5 to +5 via 0). The sentiment of these tweets were determined on the basis of the intended meaning of their textual content, including their hashtags . This gold standard data was used to provide test questions in the CrowdFlower task so as to detect and disqualify scammers. Each tweet is annotated by 7 different users. Those scammers that are not eliminated will ultimately contribute to the weighted-mean sentiment rating of the tweets, but only in proportion to their perceived reliability on the gold-standard test questions. Thus, a scammer that eludes CrowdFlower's scammer-detection filter will only have a greatly diminished contribution to the weighted-mean sentiment of a tweet. Only the annotators that do well relative to the gold-standard questions can have a significant impact on the calculation of the weighted-mean sentiment rating of the 8000 tweets.

Scammer detection:

To detect the scammers who have bypassed the initial security detection process, the following post-processing is performed. Following the potential trends of scammers we found two types of scammers: 

1. Those who choose the same answer for each question 
2. Those who choose a random answer for each question

Type 1 scammers are easily detected and successfully discarded. To detect type 2 scammers, we estimated the reliability factor for each on the basis of their annotations of gold standard data (obtained from our trial data). A standard deviation is calculated for each gold standard tweet annotation. A reliability factor is calculated by taking the ratio of the number of annotations falling in the range of standard deviation and the number of annotations performed by the user. After checking the reliability factor manually, a threshold value was selected to eliminate remaining potential scammers. 

Data format:

The data are stored in the following format: a spreadsheet with 2 fields – tweeter id, and mean annotation of each tweet (from 7 annotators). For example:

472120742755447000	-1	[I just love how my mom ALWAYS answers her phone. #not]

As per Twitter policy, the actual text of the tweet is not included in each case. It is provided above for illustration purposes only. Download the script to access the text of each tweet on an id-by-id basis.

Dataset Analysis:

Sentiment analysis of user generated data is quite challenging, even for human. Annotators with different background have annotated with different sentiment. We have calculated the standard deviation for the annotations offered by our CrowdFlower annotators.

To calculate the weighted sentiment mean, the reliability factor of each user is used as the weight value.  The mean is calculated by using this formula:

                 		Σ R(Uj) * A(Uij)
	S(ti) =       -------------------------
				         Σ R(Uj)	

S(tj) =  Sentiment of the tweet
R(Uj) = Reliability of the user j 
A(Uij) =  Annotation of user j for tweet i.

Most of the sarcastic, ironical and metaphorical data are mostly negative in nature.