HG8011: Projects 1 and 2

These are the linked projects 1 and 2 for Semantics.

Project 1 (Individual: 20%) pass/fail

• If you have not yet done so, read all of chapter 1.
• Tag all words that need to be tagged in the sentences assigned to you.
  You can find your name in the list below
  • For each word you should chose one of the senses in wordnet (Fellbaum 1998) or indicate why this is not possible.
  • If you think it is marked for sentiment, you should note its polarity and strength
  • There is a quick start guide.
  • More detailed instructions are given in the Detailed Tagging Guidelines
  • You can check your progress here.
  • The tool works best under Chrome or Firefox.
    If you log in but it does not remember you, please use incognito mode! There is an issue with cookies that we have not been able to fix.
  • You will be assessed on
    • Completion: is everything tagged
    • Accuracy: are the tags reasonable (do they agree more than a threshold with an expert annotation)
    • Your assigned sentences are here
    • If you have tagged everything reasonably accurately then you will pass
• Estimated time 3-5 hours (but it may take longer). Don't overthink things.
• You should have annotated every word by the due date. It is important you make this deadline so that we can merge the data for part 2.
• Comment words where you are not sure of the tag
• The due date is on the main page

Project 2 (Group: 20%)

• After the task is completed, you will be given a comparison of your tags with those of other annotators and a merged corpus with the majority tags tagged.
• You should discuss any differences as a group (see your group).
• You should re-tag any words on which there was not a majority agreement or on which you changed your mind.
• You should add comments for
  • Anything marked 'e', with suggestions as to how to fix the corpus
  • Anything marked 'w', with suggestions as to how to fix wordnet
    • If you are doing Czech and there are many things marked with 'w', then pick a sample: some frequent, some rare, preferably some multi-word expressions
    • Try to cover all words where you could not find a suitable synset in English
    • For words where you could find a synset, if there is nothing interesting to add, you can just list them
      • on = 77000046-n
      • právě = 00205052-r
  • If the same word appears many times with the same issue, you only need to comment on the first occurence
• Note that for some of you, the final annotator is a naive computer — it just tags most frequent sense (mfs)
  • mfs is calculated from the semcor corpus and some Sherlock Holmes Stories (DANC, SPEC, REDH, SCAN and HOUN) weighted 1:2 to normalize frequencies
  • Unseen proper nouns are tagged as per
  • Unseen closed class words are tagged as x
  • Unseen monosemous words are tagged with their single sense
  • Other unseen words are tagged None
  • If there are two or more equally frequent senses for a lemma then it is tagged None
• So feel free to over-ride them!
• Estimated time for re-annotation is 2-3 hours. Don't overthink things.
• Then write a short (6-8 page) paper:
  • In the write up you should describe the strengths and weaknesses of using a lexical resource such as wordnet to define word meaning
    • Are the senses in wordnet too coarse, too fine or just right? Justify your position.
  • You should give concrete examples from the text you analyzed. Some things you could discuss include:
    • Were some words easier or harder to annotate than others?
      • e.g. verbs, multiword expressions, concrete nouns, …
    • In cases where you disagreed with other annotators, on reflection, do you think: you were right; they were right; the definition is bad; or is there some other reason?
  • For words with senses missing in wordnet, you should write a comment with enough information to create a new entry for them consisting of, at minimum, a definition, a relational link to an existing synset and an example. E.g.
    • Lemma: arrow
    • Def: To assign a task to someone. Generally used only if the task is unpleasant or boring.
    • Ex: They come and arrow me type their document
    • Hyponym of: delegate (02391803-v)
    Don't actually create a new synset (with the edit or add a new synset button), just write in the comments (and paper).
  • You don't need an extensive literature review, but you should read and cite the references below and if you consult other lexicons (which you are encouraged to do) then you should cite them. You should also cite the wordnet you used and the story you tagged.
  • You should discuss how long it took you to do the annotation, and if you think there would be ways to make the task quicker or easier.
  • Submit softcopy (via moodle). Only one person from each group needs to submit.
  • The paper should be six to eight pages, excluding references.
    • You should not include any appendices: everything should fit within the paper.
    • References be on an extra page (page 9+), although you probably don't need many: minimally just the wordnet you are using.
  • The deadline is on the main page..
  • You should include this table at the end of the paper, on its own page:

    Distribution of Work and Time Taken

    Group	Student	A	B	C	D	E	Phase1 (h)	Phase2 (h)
    A	Good Student	--	10	10	10		3	3
    B	Speedy One	10	--	10	0		2	2.5
    C	Over Thinker	10	10	--	0		10	4
    D	Slacker McSlackface				--		0	0
    E

    Each person rates how much work each person did from 1 to 10 (with the default being 10, if someone did nothing it is 0)

    Time taken is the time in hours it took you to annotate (phase 1) and discuss and re-annotate (phase 2), not including the time for writing up the results.

    In the example given there were 4 students (A-D) and one (D) did not work at all. Student C spent more time than they should have on phase 1.

  Rubric

  You will be marked 50% on the annotation: (i) completeness (did you annotate every word), (ii) accuracy (did you select an appropriate meaning) and (iii) recall (did you only annotate words for sentiment that are not neutral) and (iv) quality of explanation (did you provide an informative explanation for every word you tagged as ‘e’ or ‘w’). The remaining 50% will be on the write up. Criteria to grade the write up include 1) language and style of the paper, 2) quality of selection of examples, 3) quality of the analysis of the data, 4) quality of discussion and conclusions, 5) overall organization and unity.

  References:

  Canonical Citation:

  Francis Bond, Andrew Devadason, Melissa Rui Lin Teo, and Luís Morgado da Costa. 2021. Teaching Through Tagging — Interactive Lexical Semantics. In Proceedings of the 11th Global Wordnet Conference, pages 273–283, University of South Africa (UNISA). Global Wordnet Association.

  Liling Tan and Francis Bond. 2012. Building and annotating the linguistically diverse NTU-MC (NTU-multilingual corpus). In International Journal of Asian Language Processing 22(4) pp 161–174.

  Christiane Fellbaum, ed. (1998) Wordnet: An Electronic Lexical Database. MIT Press

  Adam, Rambousek, Karel Pala And Aleš Horák. Sustainable long-term WordNet development and maintenance: Case study of the Czech WordNet. Cognitive Studies | Études cognitives. Warsaw: University of Silesia in Katowice & The Slavic Foundation, 2018, vol. 2018, No 18, p. 1-10. ISSN 2080-7147. DOI: 10.11649/cs.1715.

  Other References:
  

  Francis Bond, Shan Wang, Eshley Huini Gao, Hazel Shuwen Mok, and Jeanette Yiwen Tan. 2013. Developing parallel sense-tagged corpora with wordnets. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse (LAW 2013). Sofia. pp 149–158.

  Yu Jie Seah and Francis Bond. 2014. Annotation of Pronouns in a Multilingual Corpus of Mandarin Chinese, English and Japanese. In 10th Joint ACL - ISO Workshop on Interoperable Semantic Annotation Reykjavik.

  Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086.

  Shan Wang and Francis Bond. 2014. Building The Sense-Tagged Multilingual Parallel Corpus. In 9th Edition of the Language Resources and Evaluation Conference (LREC 2014), Reykjavik.

  ---

  Contributors: Francis Bond, Luís Morgado da Costa, Tuan Anh Le.

  ---

  Francis Bond <bond@ieee.org>
  Division of Linguistics and Multilingual Studies
  Nanyang Technological University
  Level 3, Room 55, 14 Nanyang Drive, Singapore 637332
  Tel: (+65) 6592 1568; Fax: (+65) 6794 6303