Lab 3

Overview

The goal of this lab is to continue development of your test suite on the one hand, and to refine your starter grammar on the other. We will be extending both to cover additional phenomena, but you may wish to revisit some of the sentences/choices from Lab 2. Again, you'll want to work on these two subtasks in parallel, though they are described separately in the instructions below.

Back to top

Test Suite

The first task is to create positive and negative example sentences illustrating the following phenomena, to the extent that they are relevant for your language:

• tense/aspect
• agreement
• negation
• argument optionality
• demonstratives/definiteness

Before you start, read the general instructions for testsuites and the formatting instructions.

Back to top

Starter grammar

The second task is to update your starter grammar by filling out the required sections of the Grammar Matrix customization questionnaire. The goal here is to get as much coverage as you can over your test suite using only the customization system (no hand-editing of tdl files yet). In particular, you'll need to address these sections:

• Tense and aspect
• Word order (if you need to add auxiliaries)
• Argument optionality
• Sentential negation
• Lexicon:
  • Rules for agreement affixes
  • Rules for tense/aspect markers
  • Rules for any other affixes that are required so you get fully inflected forms
  • Addition of auxiliaries if necessary

Begin by uploading your choices file from lab 2, and modify from there. Ask lots of questions!

NB: The sentential negation section is under development and might have more than their usual share of bugs. So start early so we have a chance to work that out.

I will try to get feedback on lab2 to you quickly. Please read that feedback for any other suggestions on how to improve your choices file.

Back to top

Make sure you can parse individual sentences

Once you have created your starter grammar (or each time you create one, as you should iterate through grammar creation and testing a few times as you refine your choices), try it out on a couple of sentences interactively to see if it works:

1. Load the grammar into the LKB.
2. Using the parse dialog box (or 'C-c p' in emacs to get the parse command inserted at your prompt), enter a sentence to parse.
3. Examine the results. If it does parse, check out the semantics (pop-up menu on the little trees). If it doesn't look at the parse chart to see why not.
4. Problems with lexical rules and lexical entries often become apparent here, too: If the LKB can't find an analysis for one of your words, it will say so, and (obviously) fail to parse the sentence.

Note that the questionnaire has a section for test sentences. If you use this, then the parse dialog will be pre-filled with your test sentences.

Back to top

[incr tsdb()] profile

The final step for this lab is to use the [incr tsdb()] grammar profiling system to test the performance of your starter grammar over your test suite, and then examine the results. (You may find in doing so that you want to refine certain aspects of your starter grammar. You can do this by uploading the file "choices" which comes with your grammar into the customization system and then tweaking from there.)

Create a test suite skeleton

• From last week, you sould already have a tsdb directory with subdirectories skeletons and home.
• The skeletons subdirectory should have files Relations and Index.lisp.
• Use make_item.pl to create a new item file from your current testsuite.
• You can replace skeletons/lab2/item with this new file, or create a new subdirectory in skeletons called lab3. In that case, you'll need to copy skeletons/Relations to skeletons/lab3/relations, put your new item in skeletons/lab3/, and edit Index.lisp to add a line for the lab3 skeleton.

Create and run an initial test suite instance

1. Start the lkb
2. Load your starter grammar. (The script file is in matrix/lkb/script.)
3. Start [incr tsdb()] (within emacs, that's M-x itsdb)
4. In the [incr tsdb()] podium, select Options > Database Root and input the path to tsdb/home.
5. In the [incr tsdb()] podium, select Options > Skeleton Root and input the path to tsdb/skeletons.
6. Optional: For future use, you can set these variables ahead of time in a file called .tsdbrc in your home directory. It should contain these lines, with path names edited appropriately:

   (in-package :tsdb)
   (setf *tsdb-home* "path-to-tsdb/home")
   (setf *tsdb-skeleton-directory* "path-to-tsdb/skeletons")
   

7. In the [incr tsdb()] podium, select File > Create. You should see your test suite in the menu there. Select it, and get a test suite instance. Post to GoPost if this doesn't work.
8. Make sure your grammr is loaded into the LKB.
9. Once you have a test suite instance, select it (by clicking on it), then do Process > All Items.
10. Explore the results, with functions such as Browse > Results and Analyze > Competence.
11. Be sure to save (i.e., not overwrite or delete) this test suite instance, as you'll be asked to turn it in.

Back to top

Write up

Your write up should include the following:

1. Documentation the new or revised choices you made in the customization system, illustrated with examples from your test suite. (Diff your lab2 and lab3 choices files to make sure you've caught all the changes.) Here's an example of what this should look like.
2. Descriptions of any properties of your language illustrated in your test suite but not covered by your starter grammar and/or the customization system.
3. Documentation the coverage of your grammar over the testsuite. If there are examples that thare parsed incorrectly (unanalyzed grammatical examples, analyzed ungrammatical examples, or grammatical examples assigned surprising parses), reflect on why that might be.
4. Finally, if there are any places where the customization system seems unable to cope with the properties of your language (within the phenomena addressed in this lab), describe them here.

Back to top

Submit your assignment

• Be sure your write up and the text-file version of your test suite are included in your grammar directory.
• Likewise, make sure to include your most current tsdb profile in the grammar directory (ideally inside tsdb/home/).
• Create a tarball:

        tar czf yourname.tgz yourgrammar
      

• Email the tarball to bond@ieee.org.

Back to top

Back to course page

Course materials borrow heavily from Linguistics 567: Knowledge Engineering for NLP at the University of Washington. Thanks to Emily Bender for letting us use them.