9. Tokenization

Tokenization is the process of breaking down text into smaller units, called tokens.
Tokens will typically approximate words, but can be sub-word units.

Typically punctuation and symbols will also be tokens
This step is foundational in NLP, allowing algorithms to handle text systematically.
Proper tokenization is crucial, as it directly affects the performance of NLP models.
Because it requires language specific information (and may not have one correct answer) we use statistical language models
We will use spaCy a free, open-source library for advanced Natural Language Processing (NLP) in Python.
- Read spaCy 101: Everything you need to know at least up to the end of the section on Linguistic Annotations
- Watch the video Text Analysis with Python: Intro to Spacy
spaCy has models for many languages, but not all the languages we want. Note that these models are not small, and may take a long time to load on a computer with limited memory.
- So we will also use models from the Universal Dependency Project
- These models are part of UDPipe a trainable pipeline for tokenization, tagging, lemmatization and dependency
- These are brought together in the spacy-udpipe package
Another toolkit that looks useful is Trankit a light-weight Transformer-based Python Toolkit for multilingual Natural Language Processing (NLP)

spaCy is included in colab by default, you don't need to install it
You must download a model before you can use it.
⚠ Then you must restart to reload dependencies
You only have to do this once!
So do this in one cell, then restart (Runtime|Restart)
import spacy spacy.cli.download("en_core_web_sm") #download English model
Or any other models you want! Generally, use a small model _sm, larger models are more accurate but much slower.
For spacy-udpipe, it is external so you need to install it with %pip install spacy-udpipe
Then you can load your models. Note that you do not have to restart anything.
The list of supported languages and models is here
%pip install spacy_udpipe import spacy_udpipe spacy_udpipe.download("en") # download English model

Load appropriate models for the following languages and then process with spacy
testdata = { "zh": "有时，敏捷的棕色狐狸跳过了懒惰的猫。多么令人兴奋啊！", # Chinese "cs": "Někdy, rychlá hnědá liška přeskočí línou kočku. Jak vzrušující!", # Czech "ja": "時々、素早い茶色の狐が怠けた猫を飛び越える。なんて素晴らしい！", # Japanese "ko": "때때로, 빠른 갈색 여우가 게으른 고양이를 뛰어넘는다. 정말 신난다!", # Korean "id": "Kadang-kadang, rubah cokelat cepat melompati kucing malas. Betapa seru!", # Indonesian "en": "Sometimes, the quick brown fox jumps over the lazy cat. How exciting!" # English }
Loop through the sentences (using doc.sents)
Loop through the tokens in each sentence and display
- word
- part-of-speech
- lemma
- morphology
See how different the morphology is for each language!

Use NLTK to download the Universal Declaration of Human Rights in all of the languages we looked at above and at least one more.
Hint:
%pip install nltk import nltk nltk.download('udhr2') # list the translations print(nltk.corpus.udhr.fileids()) # get the text of a translation text = udhr.raw('Japanese_Nihongo-UTF8').
You will have to match the translations with the language code yourself!
Count how many sentences there are in each language
Count how many times each part of speech is used in each language
Print out the results as a table
Pick one POS that is surprising to you and look at some (or all examples) for a couple of languages. For example, why are adjectives so rare in CJK?

There are many packages to help with processing language
They are all slightly different, so you have to get used to reading the manual (and searching on google or asking a LLM).
Typically you will have to do a bit of weird manual work to make everything fit together (like looking for the languages you want in a list of file ids)
Some packages work with many languages
Some are designed to work for just one language, and will often be more accurate, or have more features
It is worth looking around a little before you program something yourself to see it already exists.