Sentiment Annotation - Quick Start Guide

First we will discuss the sentiment rating scale and then the annotation process.

1) Sentiment Rating Scale and Examples

1.1 - The Scale

The Sentiment Rating tool has a scale that ranges from -100 to 100 (negative to positive sentiment).

To facilitate annotation, there are also seven (7) main Steps on the scale. These Steps are displayed on the tool itself.

Each Step corresponds to a certain score on the -100 – 100 scale:

Step	1	2	3	4	5	6	7
Score	-95	-64	-34	0	34	64	95
	← More Negative			Neutral	More Positive →

As you can see, the lowest and highest score you can give using using the 7-step scale is -95 (1) and 95 (7) respectively.
You can use the mouse or the keyboard arrow keys (← and →) to adjust it to the maximum on either side (-100, or 100).

In most cases, the 7-step scale should be enough for your sentiment annotation task, and you will not need to make fine adjustments.

1.2 - Some Examples

The table below shows some examples and the score/rating that might be assigned to them.

Score	Step	Example	From corpus	Compositional
95	7	fantastic	perfect, splendidly	very good
64	6	good	soothing, pleasure
34	5	ok	easy, interesting	sort of good, not bad
0	4	bread	puff, dig, top-knot	sliced bread
-34	3	poor	rumour, cripple	kind of bad, not good
-64	2	bad	hideous, death
-95	1	awful	deadly, foul play, horror-stricken	very bad

The compositional examples are for illustrative purposes, this guide does not show you how to tag chunks consisting of multiple concepts.

---

2) Sentiment Annotation

This section will walk you through the process of using the sentiment rating tool.

This guide assumes you already know how to log-in and that you are familiar with the general layout of the annotation interface.

You can check out the annotation Quick Start Guide for Projects 1 and 2 for a refresher.

---

2.1 - Preliminaries (What Should Be Done)

You need to annotate words in the text for their sentiment. You will determine whether a word is positive or negative (polarity), to what degree it is positive or negative (magnitude), and thereafter give it a sentiment score/rating.

Unlike sense tagging, you do not have to annotate every word. Most words are neutral for sentiment.

Other points to take note:

• The sentiment score/rating should be based on the concept used to tag that word
• It will be useful to look at, and base your decision on, the definition and example(s) of the concept (ie, synset)
• Your annotation should be context independent
• As with sense annotating, do not overthink!
• Make use of the Comments box.

---

2.2 - Annotating

1. To begin, click on the word you wish to annotate for sentiment.
2. The word will be highlighted in yellow, and its available senses loaded on the right panel.
• This should already be familiar to all of you who have worked on Projects 1 and 2.
3. Unlike in previous projects, we will be using the Sentiment Rating tool towards the bottom of the screen, as illustrated below:




NOTE: Words which were tagged as "x" will not have a Sentiment Rating tool.

4. The Rating tool contains seven (7) main Steps, for scores ranging from -100 to 100.
   • 1 being Very Negative
   • 7 being Very Positive
   • 4 being Neutral
   • The other numbers {2, 3; 5, 6} being of lesser magnitudes than their extremes

See Sentiment Scale and Examples for more information on the scale and some example words and their ratings.

5. There are two ways to set the sentiment score:
   • Using the keyboard, press Z + [Step Number] and the tool's meter will be changed accordingly. (Eg: To give it a rating of 3, press Z + 3 on the keyboard)
   • Using the mouse, drag the meter's slider to where you want it to be
   • We recommend using the keyboard for coarser ratings, and then adjusting using the mouse for finer sentiment.*
   • (Alternatively, the ← and → keys can also be used for fine adjustments. You need to click on the slider — to bring it into focus — before you can use the arrow keys)

   * In most cases, the 7-step scale should be enough for your sentiment annotation task, and you will not need to make fine adjustments.

6. As illustrated above, I have selected malodorous to rate for sentiment. It is a negative word, and so I choose to give it a rating of 2.
   
   To do this, I press Z + 2 on the keyboard.
   
   The sentiment rating tool's slider shifts accordingly to 2:

7. The rating is automatically saved.
   • A confirmation message (The sentiment score of the concept [# (sid:#####)] was updated to -64 by USERNAME) should appear at the bottom of the interface to indicate the rating was performed successfully.
   • The number indicated in the confirmation message is the numerical score, not the Step. (Find out more about the sentiment scale)
8. In the text, the rating is reflected for malodorous, and it now has a red underline below its normal underline, like so: malodorous
9. You will have to click on another word to see the second underline (for sentiment).
10. Let's take a look at another word: friend
    • Friend is a rather positive word, and should therefore be given a rather positive rating.
    • Let's give it a rating of 5.
11. To do this, I press Z + 5 on the keyboard.
12. The slider shifts accordingly:

13. Again, a confirmation message should appear at the bottom of the interface to indicate the rating was performed successfully.
14. In the text, the sentiment rating is reflected for friend, and it now has a green underline below its normal underline, like so: friend

If, by accident, you gave a sentiment rating to a word that should not have a sentiment, you can reset it by changing its sentiment rating to Neutral (4) by pressing Z + 4 on your keyboard.

Avoid using the mouse to drag the slider when neutral-ising.

Neutral-ised words will no longer have the second underline.

Multi-word Expressions?

A Multi-word Expression (MWE) should be tagged as a whole for sentiment.

If you have previously tagged the individual words in an MWE with "x" (as you should!) then only the MWE itself would have a Sentiment Rating tool available for it, as shown below:

As you can see, the MWE as good as one's word is the only one with a Sentiment Rating tool. The smaller parts of the MWE (good and good word) were correctly tagged with "x".

If you see multiple Sentiment Rating tools, then it is likely the individual words or phrases within an MWE were not tagged as "x" during sense-tagging.

Fix this by retagging these smaller components with "x" (see the sense tagging Quick Start Guide on what to do for MWEs, if you need a refresher).

References:

• Francis Bond, Tomoko Ohkuma, Luis Morgado da Costa, Yasuhide Miura, Rachel Chen, Takayuki Kuribayashi, and Wenjie Wang (2016) A multilingual sentiment corpus for Chinese, English and Japanese. In Proceedings of the LREC 2016 Workshop “Emotion and Sentiment Analysis”, Portorož. pp 59–62
• Francis Bond, Arkadiusz Janz and Maciej Piasecki (2019) A Comparison of Sense-level Sentiment Scores Proceedings of the 10th Global Wordnet Conference pp 363–372