12. Summary and Review

For revision: reread chapters 1-6, redo the tutorial problems.

NLTK Chapters: 1-6

Sequences

1. Strings, lists and tuples are different kinds of sequence object, supporting common operations such as indexing, slicing, finding the length, sorting, and membership testing using in.
2. Texts can be represented in Python using lists: ['Monty', 'Python']. We can use indexing, slicing, and the len() function on lists.
   • list = ['Monty', 'Python']
   • list[0] = 'Monty'
   • list[0:1] = ['Monty']
   • len(list) = 2
3. A string is specified in Python using single or double quotes: 'Monty Python', "Monty Python".
   • string = "string"
   • string[0] = "s"
   • string[1:3] = "rin"
   • len(string) = 5
4. Strings have some special tests associated with them:
   • string.startswith('substr')
   • string.endswith('substr')
   • string.isalpha()
   • string.islower()
   • string.istitle()
   • string.isupper()
5. Strings can be split into lists: 'Monty Python'.split() gives ['Monty', 'Python'].
6. Lists can be joined into strings: '/'.join(['Monty', 'Python']) gives 'Monty/Python'.
7. A dictionary is used to map between arbitrary types of information, such as a string and a number:
   freq['cat'] = 12.
8. We create dictionaries using the brace notation:
   pos = {}
pos = {'furiously': 'adv', 'ideas': 'n', 'colorless': 'adj'}
9. A tuple consists of a number of values separated by commas:
   t = 12345, 54321, 'hello!'
10. Tuples are normally enclosed in parentheses ()
11. List comprehensions provide a concise way to create lists. The resulting list definition tends often to be clearer than lists built using those constructs. Each list comprehension consists of an expression followed by a for clause, then zero or more for or if clauses. The result will be a list resulting from evaluating the expression in the context of the for and if clauses which follow it. If the expression would evaluate to a tuple, it must be parenthesized.
    to get nouns from a tagged corpus: nouns = [ w for (w, t) in tagged_text if t.startswith('N')]
12. A string formatting expression template % arg_tuple consists of a format string template that contains conversion specifiers like %-6s and %0.2d.
    print "Tokens = %d; Types = %d; Tokens/Type = %0.2d" % (len(words), len(set(words), 1.0*len(words)/len(set(words)))

Iterating over sequences

List Methods

String Methods

String Comparison Operators

Dictionary Methods

Frequency Distributions

Conditional Frequency Distributions

Control and Program Style

1. We process each word in a text using a for statement, such as for w in t: or for word in text:. This must be followed by the colon character and an indented block of code, to be executed each time through the loop.
2. We can also loop through things using a while loop.
   while (n != 20):
3. We can also do this in a single command (a list comprehension). To derive the vocabulary, collapsing case distinctions and ignoring punctuation, we can write set([w.lower() for w in text if w.isalpha()]).
4. We test a condition using an if statement: if len(word) < 5:. This must be followed by the colon character and an indented block of code, to be executed only if the condition is true.
5. A function is a block of code that has been assigned a name and can be reused. Functions are defined using the def keyword, as in def mult(x, y); x and y are parameters of the function, and act as placeholders for actual data values.
6. A function is called by specifying its name followed by one or more arguments inside parentheses, like this: mult(3, 4), e.g., len(text1).
7. Python programs more than a few lines long should be entered using a text editor, saved to a file with a .py extension, and accessed using an import statement.
8. Python functions permit you to associate a name with a particular block of code, and re-use that code as often as necessary.
9. A function serves as a namespace: names defined inside a function are not visible outside that function, unless those names are declared to be global.
10. Some functions are not available by default, but must be accessed using Python's import statement.
11. To find out about some variable v, type help(v) in the Python interactive interpreter to read the help entry for this kind of object.
12. Python's assignment and parameter passing use object references; e.g. if a is a list and we assign b = a, then any operation on a will modify b, and vice versa.
13. The is operation tests if two objects are identical internal objects, while == tests if two objects are equivalent. This distinction parallels the type-token distinction.
14. It is good to test functions with specialized test data (unit tests) to check that they work.
    To test + we make some data test = [(1, 1, 2), (1, 0, 1), (2, 2, 4)]

     for (a, b,c ) in test:
    	if (a + b != c):
    		print "test failed: %s + %s not equal to %s" % (a, b, c) 
        

Lexical Resources

1. A word "token" is a particular appearance of a given word in a text; a word "type" is the unique form of the word as a particular sequence of letters. We count word tokens using len(text) and word types using len(set(text)).
2. We obtain the vocabulary of a text t using sorted(set(t)).
3. We operate on each item of a text using [f(x) for x in text].
4. A frequency distribution is a collection of items along with their frequency counts (e.g., the words of a text and their frequency of appearance).
5. A text corpus is a large, structured collection of texts. NLTK comes with many corpora, e.g., the Brown Corpus, nltk.corpus.brown.
6. Often you can access many views:
   • nltk.corpus.brown.raw the whole corpus as one string
   • nltk.corpus.brown.words the whole corpus tokenized into words
   • nltk.corpus.brown.sents the whole corpus tokenized into words and split into sentences
   • nltk.corpus.brown.tagged_words the whole corpus tokenized and tagged
   • nltk.corpus.brown.tagged_sents the whole corpus tokenized and tagged and split into sents
7. Some text corpora are categorized, e.g., by genre or topic; sometimes the categories of a corpus overlap each other.
8. A conditional frequency distribution is a collection of frequency distributions, each one for a different condition. They can be used for counting word frequencies, given a context or a genre.
9. WordNet is a semantically-oriented dictionary of English, consisting of synonym sets or synsets and organized into a network.

Dealing with raw text

1. In this book we view a text as a list of words. A "raw text" is a potentially long string containing words and whitespace formatting, and is how we typically store and visualize a text.
2. We can read text from a file f using text = open(f).read().
3. We can read text from a URL u using text = urlopen(u).read().
4. We can write text to a file by opening the file for writing: file = open('output.txt', 'w'), then adding content to the file: file.write("Monty Python"), and finally closing the file: file.close().
5. We can iterate over the lines of a text file using for line in open(f):.
6. Texts found on the web may contain unwanted material (such as headers, footers, markup), that need to be removed before we do any linguistic processing.
7. Tokenization is the segmentation of a text into basic units or tokens such as words and punctuation. Tokenization based on whitespace is inadequate for many applications because it bundles punctuation together with words. NLTK provides an off-the-shelf tokenizer nltk.word_tokenize().
8. Lemmatization is a process that maps the various forms of a word (such as appeared, appears) to the canonical or citation form of the word, also known as the lexeme or lemma (e.g. appear).

Regular Expressions

1. Regular expressions are a powerful and flexible method of specifying patterns. Once we have imported the re module, we can use re.findall() to find all substrings in a string that match a pattern.
2. If a regular expression string includes a backslash, you should tell Python not to preprocess the string, by using a raw string with an r prefix: r'regexp'.
3. Regular expressions use meta-characters: . ^ $ * + ? { [ ] \ | ( )
4. When backslash is used before certain characters, e.g. \n, this takes on a special meaning (newline character); however, when backslash is used before regular expression wildcards and operators, e.g. \., \|, \$, these characters lose their special meaning and are matched literally.
5. Regular expressions can be used to discover knowledge through patterns in text:
   「([non-ascii]+)」\(([ \w]+)\) finds translation pairs
   「自然言語処理 」(natural language processing)
   \b(\w+) such as (w+) finds hypernym-hyponyms
   animals such as cats, dogs and raccoons
6. We can match and then see what we matched

         m=re.match(pattern, string)
         if m:
             print ('Match found: ', m.group())
         

Regular Expression Summary

Regular Expression Predefined Sequences

Tagging and Classification

1. Words can be grouped into classes, such as nouns, verbs, adjectives, and adverbs. These classes are known as lexical categories or parts of speech. Parts of speech are assigned short labels, or tags, such as NN, VB,
2. The process of automatically assigning parts of speech to words in text is called part-of-speech tagging, POS tagging, or just tagging.
3. Automatic tagging is an important step in the NLP pipeline, and is useful in a variety of situations including: predicting the behavior of previously unseen words, analyzing word usage in corpora, and text-to-speech systems.
4. Some linguistic corpora, such as the Brown Corpus, have been POS tagged.
5. A variety of tagging methods are possible, e.g. default tagger, regular expression tagger, unigram tagger and n-gram taggers. These can be combined using a technique known as backoff.
6. Taggers can be trained and evaluated using tagged corpora. Useful features include:
   • Tag Frequency
   • Context: previous word or tag
   • Word Shape: prefix, suffix, capitalization, character type
7. Backoff is a method for combining models: when a more specialized model (such as a bigram tagger) cannot assign a tag in a given context, we backoff to a more general model (such as a unigram tagger).
8. Part-of-speech tagging is an important, early example of a sequence classification task in NLP: a classification decision at any one point in the sequence makes use of words and tags in the local context.
9. N-gram taggers can be defined for large values of n, but once n is larger than 3 we usually encounter the sparse data problem; even with a large quantity of training data we only see a tiny fraction of possible contexts.
10. Modeling the linguistic data found in corpora can help us to understand linguistic patterns, and can be used to make predictions about new language data.
11. Supervised classifiers use labeled training corpora to build models that predict the label of an input based on specific features of that input.
12. Supervised classifiers can perform a wide variety of NLP tasks, including document classification, part-of-speech tagging, sentence segmentation, dialogue act type identification, and determining entailment relations, and many other tasks.
13. Decision trees are automatically constructed tree-structured flowcharts that are used to assign labels to input values based on their features. Although they're easy to interpret, they are not very good at handling cases where feature values interact in determining the proper label.
14. Most of the models that are automatically constructed from a corpus are descriptive they let us know which features are relevant to a given patterns or construction, but they don't give any information about causal relationships between those features and patterns.

Empirical Linguistics and Language Engineering

1. It is important to evaluate your results against (i) a baseline and if possible (ii) a gold standard.
   • The standard baseline for most tagging tasks is MFT (most frequent tag), equivalent to the unigram tagger.
   • The equivalent baseline for Word Sense Disambiguation is normally called first sense, as senses are normally ordered with the most salient (≈ most frequent) first.
2. We normally have to trade off recall (coverage) vs precision (accuracy).
   We can get more results OR better ones.
3. When training a supervised classifier, you should split your corpus into three datasets: a training set for building the classifier model; a dev-test set for helping select and tune the model's features; and a test set for evaluating the final model's performance.
4. When evaluating a supervised classifier, it is important that you use fresh (unseen) data, that was not included in the training or dev-test set. Otherwise, your evaluation results may be unrealistically optimistic.

That’s all folks

HG251: Language and the Computer Francis Bond, 2011.