Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and human language. One of the fundamental tasks in NLP is text normalization, which includes converting words into their base or root forms. This process is essential for various applications such as search engines, text analysis, and machine learning. Lemmatization and stemming are two common techniques used for this purpose. This guide explores the differences between these two techniques, their approaches, use cases, and applications, and provides example comparisons.

What is Lemmatization?

Lemmatization is the process of reducing words to their base or dictionary form, known as the lemma. This technique considers the context and the meaning of the words, ensuring that the base form belongs to the language’s dictionary. For example, the words “running,” “ran,” and “runs” are all lemmatized to the lemma “run.”

How Lemmatization Works?

Lemmatization involves several steps:

1. Part-of-Speech (POS) Tagging: Identifying the grammatical category of each word (e.g., noun, verb, adjective).
2. Morphological Analysis: Analyzing the structure of the word to understand its root form.
3. Dictionary Lookup: Using a predefined vocabulary to find the lemma of the word.

For example, the word “better” would be lemmatized to “good” if it is identified as an adjective, whereas “running” would be lemmatized to “run” if identified as a verb.

Techniques in Lemmatization

1. Rule-Based Lemmatization: Uses predefined grammatical rules to transform words. For instance, removing the “-ed” suffix from regular past tense verbs.
2. Dictionary-Based Lemmatization: Looks up words in a dictionary to find their base forms.
3. Machine Learning-Based Lemmatization: Employs machine learning models trained on annotated corpora to predict the lemma of a word.

Advantages and Disadvantages of Lemmatization

Benefits:

• Accuracy: Lemmatization provides more accurate results because it considers the context and meaning of words.
• Standardization: Ensures words are reduced to their dictionary form, aiding in tasks like text normalization and information retrieval.

Limitations:

• Complexity: Requires more computational resources and a comprehensive dictionary.
• Dependency on POS Tagging: Requires accurate POS tagging, which adds to the processing overhead.

What is Stemming?

Stemming is a more straightforward process that cuts off prefixes and suffixes (i.e., affixes) to reduce a word to its root form. This root form, known as the stem, may not be a valid word in the language. For example, the words “running,” “runner,” and “runs” might all be stemmed to “run” or “runn,” depending on the stemming algorithm used.

How Stemming Works?

Stemming algorithms apply a series of rules to strip affixes from words. The most common stemming algorithms include:

1. Porter Stemmer: Uses a set of heuristic rules to iteratively remove suffixes.
2. Snowball Stemmer: An extension of the Porter Stemmer with more robust rules.
3. Lancaster Stemmer: A more aggressive stemmer that can sometimes over-stem words.

For example, the words “running”, “runner”, and “runs” might all be reduced to “run” by a stemming algorithm, but sometimes it might also reduce “arguing” to “argu”.

Advantages and Disadvantages of Stemming

Benefits:

• Simplicity: Stemming is straightforward and computationally inexpensive.
• Speed: Faster processing time due to simple rules and lack of context consideration.

Limitations:

• Accuracy: Can produce stems that are not actual words, leading to less accurate results.
• Over-Stemming: Can sometimes strip too much off a word (e.g., “running” to “runn”).
• Under-Stemming: Can sometimes strip too little off a word (e.g., “running” to “run”).

Practical Implementation: Lemmatization with NLTK in Python

import nltk
from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

# Download necessary NLTK data
nltk.download('wordnet')
nltk.download('averaged_perceptron_tagger')

# Function to get POS tag
def get_wordnet_pos(word):
    tag = nltk.pos_tag([word])[0][1][0].upper()
    tag_dict = {"J": wordnet.ADJ, "N": wordnet.NOUN, "V": wordnet.VERB, "R": wordnet.ADV}
    return tag_dict.get(tag, wordnet.NOUN)

lemmatizer = WordNetLemmatizer()
word = "running"
lemma = lemmatizer.lemmatize(word, get_wordnet_pos(word))
print(f"Lemmatized word: {lemma}")

Output:

Lemmatized word: run
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!

Stemming with NLTK in Python

import nltk
from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
word = "running"
stem = stemmer.stem(word)
print(f"Stemmed word: {stem}")

Output:

Stemmed word: run

Natural Language Processing with Tokenization and Lemmatization

First, install the NLTK library

pip install nltk

Now, here is the example of code of difference between Lemmatization & Stemming:

import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.corpus import wordnet

# Download necessary NLTK data
nltk.download('punkt')
nltk.download('wordnet')
nltk.download('averaged_perceptron_tagger')

# Sample text
text = "The striped bats are hanging on their feet for best"

# Tokenize the text
words = nltk.word_tokenize(text)

# Initialize the stemmer and lemmatizer
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

# Apply stemming
stemmed_words = [stemmer.stem(word) for word in words]

# Function to get the part of speech tag for lemmatization
def get_wordnet_pos(word):
    tag = nltk.pos_tag([word])[0][1][0].upper()
    tag_dict = {"J": wordnet.ADJ, "N": wordnet.NOUN, "V": wordnet.VERB, "R": wordnet.ADV}
    return tag_dict.get(tag, wordnet.NOUN)

# Apply lemmatization
lemmatized_words = [lemmatizer.lemmatize(word, get_wordnet_pos(word)) for word in words]

# Print results
print("Original Text: ", text)
print("Tokenized Words: ", words)
print("Stemmed Words: ", stemmed_words)
print("Lemmatized Words: ", lemmatized_words)

Output:

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
Original Text:  The striped bats are hanging on their feet for best
Tokenized Words:  ['The', 'striped', 'bats', 'are', 'hanging', 'on', 'their', 'feet', 'for', 'best']
Stemmed Words:  ['the', 'stripe', 'bat', 'are', 'hang', 'on', 'their', 'feet', 'for', 'best']
Lemmatized Words:  ['The', 'strip', 'bat', 'be', 'hang', 'on', 'their', 'foot', 'for', 'best']

Lemmatization vs. Stemming: Key Differences