Implementing the AdaBoost Algorithm From Scratch - Coding

In this article, we will learn about the AdaBoost classifier and its practical implementation over a dataset.AdaBoost algorithm falls under ensemble boosting techniques, as we will discuss it combines multiple models to produce more accurate results and this is done in two phases:

Multiple weak learners are allowed to learn on training data
Combining these models to generate a meta-model, this meta-model aims to resolve the errors as predicted by the individual weak learners.

Note: For more information, refer Boosting ensemble models

What is AdaBoost

AdaBoost short for Adaptive Boosting is an ensemble learning used in machine learning for classification and regression problems. The main idea behind AdaBoost is to iteratively train the weak classifier on the training dataset with each successive classifier giving more weightage to the data points that are misclassified. The final AdaBoost model is decided by combining all the weak classifier that has been used for training with the weightage given to the models according to their accuracies. The weak model which has the highest accuracy is given the highest weightage while the model which has the lowest accuracy is given a lower weightage.

Institution Behind AdaBost Algorithm

AdaBoost techniques combine many weak machine-learning models to create a powerful classification model for the output. The steps to build and combine these models are as

Step1 – Initialize the weights

For a dataset with N training data points instances, initialize N [Tex]W_{i}[/Tex] weights for each data point with [Tex]W_{i} = \frac{1}{N} [/Tex]

Step2 – Train weak classifiers

Train a weak classifier M_kwhere k is the current iteration
The weak classifier we are training should have an accuracy greater than 0.5 which means it should be performing better than a naive guess

Step3 – Calculate the error rate and importance of each weak model M_k

Calculate rate error_rate for every weak classifier M_kon the training dataset
Calculate the importance of each model α_k using formula [Tex]\alpha_k = \frac{1}{2} \ln{\frac{1 – error_k}{error_k}}[/Tex]

Step4 – Update data point weight for each data point W_i

After applying the weak classifier model to the training data we will update the weight assigned to the points using the accuracy of the model. The formula for updating the weights will be [Tex]w_i = w_i \exp{(-\alpha_k y_i M_k(x_i))}[/Tex] . Here y_iis the true output and X_iis the corresponding input vector

Step5 – Normalize the Instance weight

We will normalize the instance weight so that they can be summed up to 1 using the formula [Tex]W_i = W_i / sum(W)[/Tex]

Step6 – Repeat steps 2-5 for K iterations

We will train K classifiers and will calculate model importance and update the instance weights using the above formula
The final model M(X) will be an ensemble model which is obtained by combining these weak models weighted by their model weights

Boosting Algorithms

Python implementation of AdaBoost

Python provides special packages for applying AdaBoost we will see how we can use Python for applying AdaBoost on a machine learning problem.

In this problem, we are given a dataset containing 3 species of flowers and the features of these flowers such as sepal length, sepal width, petal length, and petal width, and we have to classify the flowers into these species.

Import Libraries

Let’s begin with importing important libraries that we will require to do our classification task:

Python

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import AdaBoostClassifier
import warnings
warnings.filterwarnings("ignore")

Reading And Describing The Dataset

After, importing the libraries we will load our dataset using the pandas read_csv method as:

Python

# Reading the dataset from the csv file
# separator is a vertical line, as seen in the dataset
data = pd.read_csv("Iris.csv")

# Printing the shape of the dataset
print(data.shape)

(150, 6)

We can see our dataset contains 150 rows and 6 columns. Let us take a look at our actual content in the dataset using head() method as:

Python

data.head()

	Id	SepalLengthCm	SepalWidthCm	PetalLengthCm	PetalWidthCm	Species
0	1	5.1	3.5	1.4	0.2	Iris-setosa
1	2	4.9	3.0	1.4	0.2	Iris-setosa
2	3	4.7	3.2	1.3	0.2	Iris-setosa
3	4	4.6	3.1	1.5	0.2	Iris-setosa
4	5	5.0	3.6	1.4	0.2	Iris-setosa

Dropping Irrevelant Column and Separating Target Variable

The first column is the Id column which has no relevance to flowers so, we will drop it using drop() function. The Species column is our target feature and tells us about the species the flowers belong to. We will separate it using pandas iloc slicing.

Python

data = data.drop('Id',axis=1)
X = data.iloc[:,:-1]
y = data.iloc[:,-1]
print("Shape of X is %s and shape of y is %s"%(X.shape,y.shape))

Shape of X is (150, 4) and shape of y is (150,)

Unique values in our Target Variable

Since this is a classification task we will see the numbers of categories we want to classify our dataset input vector into.

Python

total_classes = y.nunique()
print("Number of unique species in dataset are: ",total_classes)

Number of unique species in dataset are: 3

Python

distribution = y.value_counts()
print(distribution)

Iris-virginica 50 Iris-setosa 50 Iris-versicolor 50 Name: Species, dtype: int64

Let’s dig deep into our dataset, and we can see in the above image that our dataset contains 3 classes into which our flowers are distributed also since we have 150 samples all three species have an equal number of samples in the dataset, so we have no class imbalance.

Splitting The Dataset

Now, we will split the dataset for training and validation purposes, the validation set is 25% of the total dataset. For dividing the dataset into training and testing we will use train_test_split method from the sklearn model selection.

Python

X_train, X_val, Y_train, Y_val = train_test_split(
    X, y, test_size=0.25, random_state=28)

Applying AdaBoost

After creating the training and validation set we will build our AdaBoost classifier model and fit it over the training set for learning. To fit our AdaBoost model we need our dependent variable y and independent variable x.

Python

# Creating adaboost classifier model
adb = AdaBoostClassifier()
adb_model = adb.fit(X_train,Y_train)

Accuracy of the AdaBoost Model

As we fit our model on the train set, we will check the accuracy of our model on the validation set. To check the accuracy of the model we will use the validation dataset that we have created using the train_test_split method.

Python

print("The accuracy of the model on validation set is", adb_model.score(X_val,Y_val))

The accuracy of the model on validation set is 0.9210526315789473

As we can see the model has an accuracy of 92% on the validation set which is quite good with no hyperparameter tuning and feature engineering.

Elevate your coding journey with a Premium subscription. Benefit from ad-free learning, unlimited article summaries, an AI bot, access to 35+ courses, and more-available only with GeeksforGeeks Premium! Explore now!

Reffered: https://www.geeksforgeeks.org

Machine Learning

Related
Grey wolf optimization - Introduction
Weibull Hazard Plot
Emotion Detection using Bidirectional LSTM
Numpy Gradient - Descent Optimizer of Neural Networks
Test of Multicollinearity

Type:	Geek
Category:	Coding
Sub Category:	Tutorial
Uploaded by:	Admin
Views:	10