Natural Language Processing (NLP) is a field constantly evolving, and a crucial component of its success is the ability to understand the structure and formation of words – morphology. The Python library word_forms emerges as a powerful tool in this domain, simplifying the extraction of morphological information from English words. This article delves into the technical aspects of morphology, the functionalities of the word_forms library, and practical applications in NLP tasks.

Morphology: The Foundation of Word Structure in NLP

Morphology is the study of morphemes – the smallest units of meaning in a language. Morphemes can be either free (stand-alone words) or bound (prefixes, suffixes, and infixes). The combination of morphemes forms complex words, conveying nuances of tense, plurality, and other grammatical properties.

Morphological analysis is a fundamental task in NLP that involves studying the structure of words and their components, such as roots, prefixes, and suffixes. It helps in understanding how words are formed and how they relate to each other. Morphological analysis is essential for various NLP applications, including text normalization, information retrieval, and machine translation.

Overview of the word_forms Library

The word_forms library is a Python package designed to generate all possible forms of an English word. It can conjugate verbs, pluralize nouns, and connect different parts of speech, making it a versatile tool for morphological analysis. The library is built on top of linguistic resources like WordNet and the XTAG project, ensuring accurate and comprehensive word form generation.

It utilizes the WordNet lexical database to provide morphological information, including:

• Inflections: Different grammatical forms of a word (e.g., “run,” “runs,” “running”)
• Derivations: Related words formed by adding affixes (e.g., “happy,” “happiness,” “unhappy”)
• Lemmatization: Reducing words to their base or dictionary form (e.g., “better” becomes “good”)
• Parts of Speech: Identifying the grammatical category of a word (e.g., noun, verb, adjective)

Installation and Setup

You can install word_forms in two ways: first, via PIP; and second, by installing it from the source.

Via PIP:

!pip install word_forms

From Source:

To install the library from the source, first clone the repository, and then install it by running the setup.py file.

git clone https://github.com/gutfeeling/word_forms.git

cd word_forms
python setup.py install

Core Functionalities of the word_forms Library

1. Generating Word Forms

The primary function of the word_forms library is to generate all possible forms of a given word. This includes different parts of speech such as nouns, verbs, adjectives, and adverbs. Here is an example:

from word_forms.word_forms import get_word_forms

word_forms = get_word_forms("president")
print(word_forms)

Output:

{
    'n': {'presidents', 'presidentships', 'presidencies', 'presidentship', 'president', 'presidency'},
    'a': {'presidential'},
    'v': {'preside', 'presided', 'presiding', 'presides'},
    'r': {'presidentially'}
}

2. Conjugating Verbs

The library can conjugate verbs into various tenses and forms. For example:

verb_forms = get_word_forms("run")
print(verb_forms['v'])

Output:

{'run', 'ran', 'running', 'runs'}

3. Pluralizing Nouns

The word_forms library can also pluralize singular nouns:

noun_forms = get_word_forms("cat")
print(noun_forms['n'])

Output:

{'catty', 'cattiness', 'cattinesses', 'catties', 'cat', 'cats'}

4. Connecting Parts of Speech

Another useful feature is connecting different parts of speech. For instance, you can find the adjective form of a noun:

adjective_forms = get_word_forms("politician")
print(adjective_forms['a'])

Output:

{'political'}

Practical Application and Examples with word_forms

Example 1: Building a Lemmatizer with word_forms

Lemmatization is the process of reducing a word to its base form. This process is important because it helps us understand the context of words in a text.

To build a lemmatizer using word_forms, first import the get_word_forms function from the library. Then, create a function named lemmatize that takes a word as input and returns its root form as output.

Within this function, use list comprehension to find all possible forms of the input word. Sort these forms by length and return the shortest form as the lemma. Implement a try-except block to raise an error if the word doesn’t exist.

from word_forms.word_forms import get_word_forms

def lemmatize(word):
    all_forms = [word for pos_form in get_word_forms(word).values() for word in pos_form]
    all_forms.sort()
    all_forms.sort(key=len)
    try:
        return all_forms[0]
    except IndexError:
        raise ValueError("{} is not a real word".format(word))

lemmatize('working')

Output:

work

Example 2: Different Forms of a Word Using word_forms

The get_word_forms function from the word_forms library retrieves all morphological variants of a given word. It returns the noun form as ‘n’, adjective as ‘a’, verb as ‘v’, and adverb as ‘r’ forms. Here are a few examples of using this function:

get_word_forms("president")

Output:

{'n': {'presidencies',
  'presidency',
  'president',
  'presidents',
  'presidentship',
  'presidentships'},
 'a': {'presidential'},
 'v': {'preside', 'presided', 'presides', 'presiding'},
 'r': {'presidentially'}}

get_word_forms('continent', 0.8)

Output:

{'n': {'continence',
  'continences',
  'continencies',
  'continency',
  'continent',
  'continents'},
 'a': {'continent', 'continental'},
 'v': set(),
 'r': set()}

Conclusion

The word_forms library is a valuable tool for morphological analysis in NLP. Its ability to generate all possible forms of an English word makes it useful for various NLP applications, including text normalization, information retrieval, sentiment analysis, and machine translation. While it has some limitations, ongoing improvements and contributions from the community can help address these issues and make the library even more robust.