Semantic similarity is the similarity between two words or two sentences/phrase/text. It measures how close or how different the two pieces of word or text are in terms of their meaning and context.

In this article, we will focus on how the semantic similarity between two sentences is derived. We will cover the following most used models.

1. Dov2Vec – An extension of word2vec
2. SBERT – Transformer-based model in which the encoder part captures the meaning of words in a sentence.
3. InferSent -It uses bi-directional LSTM to encode sentences and infer semantics.
4. USE (universal sentence encoder) – It’s a model trained by Google that generates fixed-size embeddings for sentences that can be used for any NLP task.

What is Semantic Similarity?

Semantic Similarity refers to the degree of similarity between the words. The focus is on the structure and lexical resemblance of words and phrases. Semantic similarity delves into the understanding and meaning of the content. The aim of the similarity is to measure how closely related or analogous the concepts, ideas, or information conveyed in two texts are.

In NLP semantic similarity is used in various tasks such as

1. Question Answering – Enhances QA system by deriving semantic similarity between user queries and document content.
2. Recommendation systems – Semantic similarity between user content and available content
3. Summarization – Helps in summarizing similar content question answering, and text matching.
4. Corpus clustering -Helps in grouping documents with similar content.

There are certain approaches for measuring semantic similarity in natural language processing (NLP) that include word embeddings, sentence embeddings, and transformer models.

Word Embedding

To understand semantic relationships between sentences one must be aware of the word embeddings. Word embeddings are used for vectorized representation of words. The simplest form of word embedding is a one-hot vector. However, these are sparse, very high dimensional, and do not capture meaning. The more advanced form consists of the Word2Vec (skip-gram, cbow), GloVe, and Fasttext which capture semantic information in low dimensional space. Kindly look at the embedded link to get a deeper understanding of this.

Word2Vec

Word2Vec represents the words as high-dimensional vectors so that we get semantically similar words close to each other in the vector space. There are two main architectures for Word2Vec:

• Continuous Bag of Words: The objective is to predict the target word based on the context of surrounding words.
• Skip-gram: The model is designed to predict the surrounding words in the context.

Doc2Vec

Similar to word2vec Doc2Vec has two types of models based on skip gram and CBOW. We will look at the skip gram-based model as this model performs better than the cbow-based model. This skip-gram-based model is called ad PV-DM (Distributed Memory Model of Paragraph Vectors).

PV-DM model

PV-DM is an extension of Word2Vec in the sense that it consists of one paragraph vector in addition to the word vectors.

• Paragraph Vector and Word Vectors: Suppose there are N paragraphs in the corpus, and M words in the vocabulary.
  • Now based on the lines of word2vec we will have M*Q (Q is the dimesnon of word embedding) matrix which will be our embedding matrix for the word embedding. Additionally, we will have an N*P matrix for our paragraphs (p is the dimension of paragraph embedding).
  • The paragraph vector is shared across all contexts generated from the same paragraph but not across different paragraphs.
  • The word vector matrix W is shared across paragraphs.
• Averaging or Concatenation: To predict the next word in a context, the paragraph vector and word vectors are combined using either averaging or concatenation.
• Distributed Memory Model (PV-DM): The paragraph token acts as a memory that retains information about what is missing from the current context or the topic of the paragraph.
• Training with Stochastic Gradient Descent: Stochastic gradient descent is used to train the paragraph vectors and word vectors. The gradient is obtained via backpropagation. During each step of stochastic gradient descent, a fixed-length context is sampled from a random paragraph, and the error gradient is computed to update the model parameters.
• Inference at Prediction Time: Once the model is trained the paragraph vectors are discarded and only the word vectors and softmax weights are retained.
  • For finding the paragraph vector of new text During prediction time, an inference step is performed. This is also achieved through gradient descent. In this step, the word vectors (W) and softmax weights, are fixed while the paragraph vector is learned through backpropagation.
  • For two sentences for which we need to find the similarity the two paragraph vectors are obtained and based on the similarity between the two vectors the similarity between the sentences is obtained

In summary, the algorithm itself has two key stages:

• Training to get word vectors W, softmax weights U, b, and paragraph vectors D on already seen paragraphs.
• The inference stage is to get paragraph vectors D for new paragraphs (never seen before) by adding more columns in D and gradient descending on D while holding W, U, and mixed.

We use the learned paragraph vectors to predict some particular labels using a standard classifier, e.g., logistic regression.

Python Implementation of Doc2Vec

Below is the simple implementation of Doc2Vec.

1. We first tokenize the words in each document and convert them to lowercase.
2. We then create TaggedDocument objects required for training the Doc2Vec model. Each document is associated with a unique tag (document ID). This is the paragraph vector.
3. The parameters (vector_size, window, min_count, workers, epochs) control the model’s dimensions, context window size, minimum word count, parallelization, and training epochs.
4. We then infer a vector representation for a new document that was not part of the training data.
5. We then calculate the similarity score.

Output:

Original Document: The baby was laughing and palying
Document 2: Similarity Score: 0.9838361740112305
Document Text: The baby learned to walk in the 5th month itself

Document 0: Similarity Score: 0.9455077648162842
Document Text: The movie is awesome. It was a good thriller

Document 1: Similarity Score: 0.8828089833259583
Document Text: We are learning NLP throughg GeeksforGeeks

SBERT

SBERT adds a pooling operation to the output of BERT to derive a fixed-sized sentence embedding. The sentence is converted into word embedding and passed through a BERT network to get the context vector. Researchers experimented with different pooling options but found that at the mean pooling works the best. The context vector is then averaged out to get the sentence embeddings.

SBERT uses three objective functions to update the weights of the BERT model. The Bert model is structured differently based on the type of training data that drives the objective function.

1. Classification objective Function

• This model architecture uses part of a sentence along with labels as training data.
• Here the Bert model is structured as a siamese network. What is the Siamese netowrk? It consists of two identical subnetworks each of which is a BERT model. The two models share/have the same parameters/weights. Parameter updating is mirrored across both sub-models. On the top of the polling layer, we have softmax classifier with the number of nodes the as number of labels the in-training data.
• The sentences are passed together to get sentence embeddings u and v along with element-wise different |u-v|. These three vectors (u, v,|u-v|) are multiplied with a weight vector W of size (3n*K) to get a softmax classification.
  
  Here,
  • n is the dimension of the sentence embeddings
  • k the number of labels.
• The optimization is performed using cross-entropy loss.

SBERT with Classification Objective Function

2. Regression Objective function

This also uses the pair of sentences with labels as training data. The network is also structured as a Siamese network. However, instead of the softmax layer the output of the pooling layer is used to calculate cosine similarity and mean squared-error loss is used as the objective function to train the BERT model weights.

SBERT with Regression Objective Function

3. Triplet objective function

Here the model is structured as triplet networks.

• In a Triplet Network, three subnetworks process an anchor sentence, a positive (similar) sentence, and a negative (dissimilar) sentence. The model learns to minimize the distance between the anchor and positive sentences while maximizing the distance between the anchor and negative sentences.
• To train the model we need a dataset that has an anchor dataset a, a positive sentence p, and a negative sentence n. An example of such a data set is ‘The Wikipedia section triplets dataset’.

Mathematically, we minimize the following loss function.

• with sa, sp, sn the sentence embedding for a/n/p,
• || · || a distance metric and margin.
• Margin ϵ ensures that sp is at least closer to sa than sn.

Python Implementation

To implement it first we need to install Sentence transformer framework

!pip install -U sentence-transformers

• The SentenceTransformer class is used to load an SBERT model.
• The SentenceTransformer class is used to load an SBERT model.
• We use the scipy cosine distance to calculate the distance between two vectors. To get similarity we subtract it from 1

Output:

Test sentence: I liked the movie.

For The movie is awesome. It was a good thriller
Similarity Score = 0.682051956653595 

For We are learning NLP throughg GeeksforGeeks
Similarity Score = 0.0878136083483696 

For The baby learned to walk in the 5th month itself
Similarity Score = 0.04816452041268349 

InferSent

The structure comprises two components:

Sentence Encoder

• The first is a sentence encoder responsible for receiving word vectors and transforming sentences into encoded vectors.
• InferSent starts with pre-trained word embeddings. Words are embedded into continuous vector representations.
• These embeddings serve as the input to the bi-directional LSTM that is capable of capturing sequential information and dependencies in data.
• After processing the input sentence through the bi-directional LSTM, a pooling layer is applied to obtain a fixed-size vector representation of the entire sentence. Common pooling techniques include max pooling, mean pooling, or concatenation of the final hidden states.

Classifier

• The pooled sentence representation is fed through one or more fully connected layers.
• These layers serve as the classifier part whick that takes the pooled encoded vectors as input and generates a classification, identifying whether the relationship between the sentences is entailment, contradiction, or neutral.
• 3 matching methods are applied to extract relations between pair of sentences u and v.
  • concatenation of the two representations (u, v)
  • element-wise product u * v
  • element-wise difference |u — v |
• The resulting vector is fed into a 3-class classifier (entailment, contradiction, and neutral) consisting of multiple fully connected layers followed by a SoftMax layer.

Training Sentence Encoder for Classification

Python Implementation

Implementing an infersent model is a bit lengthy process as there is no standard hugging face API available. Infersent comes traiend in two version. Version1 is trained with Glove and Version2 is trained with FastText wordembedding. We will use version 2 as it takes less time to download and process.

First, we need to build the infersent model. The below class does that. It has been sourced from Python Zoo model.

Next, we download the state-of-the-art fastText embeddings and download the pre-trained models.

• The code downloads the Punkt tokenizer model from NLTK, which is used for tokenization.
• We then set the path (MODEL_PATH) to the pre-trained InferSent model file and loads its weights into an instance of the InferSent class.
• We specify the path to our word embeddings(W2V_PATH)
• The build_vocab_k_words method is called to load the embeddings for the K most frequent words (in this case, the top 100,000 words).

We then use the model for inference

Output:

Test Sentence: I liked the movie

For The movie is awesome. It was a good thriller
Similarity Score = 0.5299297571182251 

For We are learning NLP throughg GeeksforGeeks
Similarity Score = 0.33156681060791016 

For The baby learned to walk in the 5th month itself
Similarity Score = 0.20128820836544037

USE – Universal Sentence Encoder

At a high level, it consists of an encoder that summarizes any sentence to give a sentence embedding which can be used for any NLP task.

The encoder part comes in two forms and either of them can be used

• Transformer – Here the encoder part of the original transformer architecture is used. The architecture consists of 6 stacked transformer layers. Each layer has a self-attention module followed by a feed-forward network. The output context-aware word embeddings are added element-wise and divided by the square root of the length of the sentence to account for the sentence-length difference. We get a 512-dimensional vector as output sentence embedding.
• Deep averaging network- the embeddings for word and bi-grams present in a sentence are averaged together. Then, they are passed through a 4-layer feed-forward deep DNN to get 512-dimensional sentence embedding as output. The embeddings for word and bi grams are learned during training.

Training of the USE

The USE is trained on a variety of unsupervised and supervised tasks such as Skipthoughts, NLI, and more using the below principles.

• Tokenize the sentences after converting them to lowercase
• Depending on the type of encoder, the sentence gets converted to a 512-dimensional vector
• The resulting sentence embeddings are subsequently used to update the model parameters.
• The trained model is then once again applied to produce a fresh 512-dimensional sentence embedding.

Training of Encoder

Python Implementation

We load the Universal Sentence Encoder’s TF Hub module.

• module_url contains the URL to load the Universal Sentence Encoder (version 4) from TensorFlow Hub.
• The hub. load function is used to load the Universal Sentence Encoder model from the specified URL (module_url)
• We define a function named embed that takes an input text and returns the embeddings using the loaded Universal Sentence Encoder model.

We draw the similarity score on our sample data below

Output

Test Sentence: ['I liked the movie very much']

For ['The movie is awesome and It was a good thriller']
Similarity Score = 0.6519516706466675 

For ['We are learning NLP throughg GeeksforGeeks']
Similarity Score = 0.06988027691841125 

For ['The baby learned to walk in the 5th month itself']
Similarity Score = -0.01121298223733902 

Conclusion

In this article we understood semantic similarity and its application. We saw the architecture of top 4 sentence embedding models used for semantic similarity calculation of sentences and their implementation in python.