XGBoost is a powerful machine-learning library that efficiently implements gradient boosting. It is widely used for its performance and flexibility. In XGBoost, two key parameters that often come up during model training are objective and feval. Understanding their differences is crucial for effectively leveraging the library’s capabilities.

Overview of XGBoost

Before diving into the specifics of objective and feval, it’s important to understand the basic concepts of XGBoost. XGBoost (eXtreme Gradient Boosting) implements gradient-boosted decision trees designed for speed and performance.

• High performance and scalability
• Flexibility through parameter tuning
• Robust handling of missing values
• Support for parallel and distributed computing

The objective Parameter in XGBoost

The objective parameter in XGBoost specifies the learning task and the corresponding loss function to be minimized. It defines the type of prediction problem being solved (e.g., regression, classification). Below are the some Common Objectives.

• reg:squarederror: Regression with squared loss
• reg:logistic: Logistic regression for binary classification
• binary:logistic: Binary classification with logistic loss
• multi:softmax: Multiclass classification using the softmax objective
• multi:softprob: Multiclass classification with probability output
• rank:pairwise: Learning to rank with pairwise loss

The feval Parameter

The feval parameter in XGBoost allows you to define a custom evaluation metric. This is particularly useful when the default evaluation metrics provided by XGBoost do not meet the specific needs of your problem. The custom evaluation function specified in feval should take two arguments: preds (predictions) and dtrain (the training data). It should return a list containing the evaluation metric’s name and value.

To explain the difference between objective and feval in xgboost, we’ll create a simple binary classification example using the built-in iris dataset. We’ll convert it to a binary classification problem by selecting only two classes, and then we’ll define a custom evaluation metric in R Programming Language.

Step 1: Prepare the Data

First, we’ll prepare the dataset by selecting only two classes from the iris dataset and splitting it into training and testing sets.

# Load necessary libraries
library(xgboost)
library(caret)

# Load the iris dataset
data(iris)

# Convert to binary classification problem by selecting two classes
binary_iris <- subset(iris, Species %in% c("setosa", "versicolor"))
binary_iris$Species <- as.integer(binary_iris$Species == "versicolor")

# Split the data into training and testing sets
set.seed(123)
trainIndex <- createDataPartition(binary_iris$Species, p = 0.7, list = FALSE)
train_data <- binary_iris[trainIndex, ]
test_data <- binary_iris[-trainIndex, ]

# Convert to DMatrix
dtrain <- xgb.DMatrix(data = as.matrix(train_data[, -5]), label = train_data$Species)
dtest <- xgb.DMatrix(data = as.matrix(test_data[, -5]), label = test_data$Species)

Step 2: Define Parameters and Train with objective

We’ll define the parameters for the xgboost model, including the objective parameter for binary logistic regression.

# Define parameters
params <- list(
  objective = "binary:logistic",  # Binary classification
  max_depth = 3,
  eta = 0.1,
  nthread = 2
)

# Train the model
bst <- xgb.train(params = params, data = dtrain, nrounds = 50)

Step 3: Define Custom Evaluation Function and Train with feval

Now, we’ll define a custom evaluation function that calculates the error rate and use it during training.

# Define a custom evaluation function
custom_eval <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  err <- mean(as.numeric(preds > 0.5) != labels)
  return(list(metric = "custom_error", value = err))
}

# Train the model with the custom evaluation function
bst_with_feval <- xgb.train(
  params = params, 
  data = dtrain, 
  nrounds = 50, 
  feval = custom_eval, 
  maximize = FALSE
)

Step 4: Predict and Evaluate

Finally, we’ll use the trained model to make predictions on the test set and evaluate the performance using the built-in confusion matrix from the caret package.

# Predict on the test set
preds <- predict(bst, newdata = dtest)
preds_with_feval <- predict(bst_with_feval, newdata = dtest)

# Convert probabilities to binary labels
pred_labels <- ifelse(preds > 0.5, 1, 0)
pred_labels_with_feval <- ifelse(preds_with_feval > 0.5, 1, 0)

# Confusion matrix for model without custom evaluation
conf_matrix <- confusionMatrix(factor(pred_labels), factor(test_data$Species))
print(conf_matrix)

# Confusion matrix for model with custom evaluation
conf_matrix_with_feval <- confusionMatrix(factor(pred_labels_with_feval), 
                                          factor(test_data$Species))
print(conf_matrix_with_feval)

Output:

Confusion Matrix and Statistics

          Reference
Prediction  0  1
         0 15  0
         1  0 15
                                     
               Accuracy : 1          
                 95% CI : (0.8843, 1)
    No Information Rate : 0.5        
    P-Value [Acc > NIR] : 9.313e-10  
                                     
                  Kappa : 1          
                                     
 Mcnemar's Test P-Value : NA         
                                     
            Sensitivity : 1.0        
            Specificity : 1.0        
         Pos Pred Value : 1.0        
         Neg Pred Value : 1.0        
             Prevalence : 0.5        
         Detection Rate : 0.5        
   Detection Prevalence : 0.5        
      Balanced Accuracy : 1.0        
                                     
       'Positive' Class : 0          
                                     

Confusion Matrix and Statistics

          Reference
Prediction  0  1
         0 15  0
         1  0 15
                                     
               Accuracy : 1          
                 95% CI : (0.8843, 1)
    No Information Rate : 0.5        
    P-Value [Acc > NIR] : 9.313e-10  
                                     
                  Kappa : 1          
                                     
 Mcnemar's Test P-Value : NA         
                                     
            Sensitivity : 1.0        
            Specificity : 1.0        
         Pos Pred Value : 1.0        
         Neg Pred Value : 1.0        
             Prevalence : 0.5        
         Detection Rate : 0.5        
   Detection Prevalence : 0.5        
      Balanced Accuracy : 1.0        
                                     
       'Positive' Class : 0 

• The objective parameter is set to "binary:logistic" for binary classification, guiding the model’s optimization process.
• The feval parameter is used to specify a custom evaluation function, which calculates the error rate during training.
• The example demonstrates how to train an xgboost model using both the objective parameter and a custom evaluation function with feval.

This example explained how to utilize both objective and feval parameters in xgboost to tailor the model training and evaluation process according to specific needs.

Key Differences between Objective and feval in xgboost in R