Table of Content

• Understanding Word Embeddings
• What is Continuous Bag of Words (CBOW)?
• What is Skip-Gram Model?
• Key Differences Between CBOW and Skip-Gram
• Training Process for CBOW
• Advantages and Disadvantages of CBOW Model
• Use Cases and Applications of CBOW
• Training Process for Skip Gram
• Advantages and Disadvantages of Skip Gram Model
• Use-Cases and Applications of Skip Gram Model

Aspect	CBOW (Continuous Bag of Words)	Skip-Gram
Concept	Predicts a target word based on context words.	Predicts context words given a target word.
Architecture	Averages context word vectors to predict the target word.	Uses the target word vector to predict multiple context words.
Training Process	Minimizes cross-entropy loss to predict the target word.	Maximizes the likelihood of context words around a target word using techniques like negative sampling or hierarchical softmax.
Training Speed	Faster due to averaging of context vectors and fewer updates.	Slower because it predicts multiple context words, requiring more updates.
Performance with Infrequent Words	Less effective in representing rare words.	More effective, as it captures detailed word-context relationships.
Quality of Word Embeddings	Produces decent word embeddings, but not as rich as Skip-Gram.	Produces higher quality embeddings, capturing subtle semantic nuances.
Hyperparameter Sensitivity	Less sensitive to hyperparameters compared to Skip-Gram.	More sensitive, requiring careful tuning of parameters like learning rate and context window size.
Computational Resources	Requires fewer resources due to simpler training process.	Requires more computational power and memory.
Use Cases	Suitable for tasks requiring speed over detailed word representations, like text classification and sentiment analysis.	Ideal for tasks needing high-quality embeddings and detailed semantic relationships, such as word similarity tasks, named entity recognition, and machine translation.

Training Process for CBOW

This training process helps the CBOW model learn meaningful word embeddings that capture the semantic relationships between words based on their contexts in the training corpus.

• Step 1: Data Preprocessing
  • Tokenize the corpus.
  • Create word embeddings (e.g., Word2Vec).
• Step 2: CBOW Model Architecture
  • Initialize input/output layers.
  • Define the hidden layer.
• Step 3: Training the CBOW Model, For each word:
  • Encode context words into embeddings.
  • Average embeddings for the context vector.
  • Pass context vector through the network.
  • Calculate loss and update weights.
  • Repeat for multiple epochs.
• Step 4: Inference with the CBOW Model
  • Encode and average context words.
  • Pass the context vector through the network to predict the word

Implementation of CBOW: Pseudocode

# Assume we have a simple neural network with one hidden layer

for epoch in range(num_epochs):
    for context, target in training_data:
        context_vector = average(word_embeddings[context])
        predicted_word = neural_network(context_vector)
        
        loss = calculate_loss(predicted_word, target)
        backpropagate(loss)
        update_weights()

# Once trained, for inference
target_context = ['the', 'quick', 'brown']
context_vector = average(word_embeddings[target_context])
predicted_word = neural_network(context_vector)

Advantages and Disadvantages of CBOW Model

Advantages of CBOW Model

• CBOW is faster to train compared to Skip-gram, especially for large datasets.
• It tends to perform well with frequent words and is useful in scenarios where context matters more than individual word positions.

Disadvantages of CBOW Model

• CBOW might not perform as well as Skip-gram in capturing rare words or phrases.
• It doesn’t preserve word order information, which can be crucial in some applications.

Use Cases and Applications of CBOW

• Word Embeddings: CBOW is widely utilized to make word embeddings which in turn is employed in different NLP applications such as sentiment analysis, machine translation and information retrieval.
• Recommendation Systems: CBOW embeddings can be exploited for understanding words relationship for similarity which is very important in recommendations system to recommend like items or content.
• Text Classification: The result of CBOW can be used as features in the text classification tasks, when the important factor is a context of words.

Training Process for Skip Gram

Step 1: Data Preprocessing

1. Tokenize the corpus into individual words.
2. Create word embeddings using techniques like Word2Vec.

Step 2: Skip-gram Model Architecture

1. Initialize input and output layers.
2. Define a hidden layer with a specified number of neurons.

Step 3: Training the Skip-gram Model

For each word w in the training corpus:

1. Retrieve context words surrounding ????w within a specified window size.
2. For each context word:
   • Encode the target word w into its corresponding word embedding.
   • Pass the word embedding as input to the neural network.
   • Forward propagate through the network to obtain the predicted context word.
   • Compare the predicted context word with the actual context word.
   • Calculate the loss using a suitable loss function (e.g., cross-entropy).
   • Backpropagate the error to update the weights.
3. Repeat the process for multiple epochs until convergence.

Step 4: Inference with the Skip-gram Model

1. Given a target word, retrieve its word embedding.
2. Pass the word embedding through the trained neural network.
3. Obtain the predicted context words as the output of the network.

Implementation of Skip Gram: Pseudocode

#Assume we have a simple neural network with one hidden layer

for epoch in range(num_epochs):
    for target_word, context_words in training_data:
        for context_word in context_words:
            target_embedding = word_embeddings[target_word]
            predicted_context = neural_network(target_embedding)
            
            loss = calculate_loss(predicted_context, context_word)
            backpropagate(loss)
            update_weights()

# Once trained, for inference
target_word = 'apple'
target_embedding = word_embeddings[target_word]
predicted_contexts = neural_network(target_embedding)

Advantages and Disadvantages of Skip Gram Model

Advantages of Skip Gram Model

• Huge advantage of Skip-Gram is that it is able to perform better with low frequency words because the idea is actually to output as many context words as possible given the target word.
• It preserves word order information, which can be beneficial in tasks where word order matters, such as language translation or sequence generation.

Disadvantages of Skip Gram Model

• Training the Skip-Gram model can be computationally expensive compared to CBOW, especially for large datasets, due to the need to predict multiple context words for each target word.
• Skip-Gram might not perform as well with very frequent words compared to CBOW.

Use-Cases and Applications of Skip Gram Model

• Utilized in different areas of Natural Language Processing such as sentiment analysis, machine interpretation, and information retrieval.
• Semantic Similarity: Skip-Gram embeddings are useful for calculating the semantic relation between two words and such applications include recommendation systems and search engines.
• Text Generation: Skip-Gram embeddings can be effectively used in text generation task to generate meaningful sequences of words to complete the given phrase.

CBOW vs Skip Gram -FAQs

What is the main difference between CBOW and Skip-Gram?

The main difference between CBOW and Skip-Gram is the direction of prediction. CBOW predicts a target word based on its context words, while Skip-Gram predicts context words based on a target word.

Which model is better for rare words?

Skip-Gram is better for rare words, as it is less sensitive to overfitting frequent words and requires less data to achieve good performances.

Which model is faster to train?

CBOW is generally faster to train than Skip-Gram, due to its simpler prediction task.