Area Under the Curve (AUC) is a common metric used to evaluate the performance of classification models, especially in binary classification tasks. Comparing multiple AUCs can provide insights into which model performs best across different datasets or model configurations. This article provides a step-by-step guide on how to compare multiple AUCs in parallel in R Programming Language using both base R functions and specialized packages.

Understanding AUC

The Area Under the Curve (AUC) of a Receiver Operating Characteristic (ROC) curve is a measure of how well a model distinguishes between classes. A higher AUC typically indicates better predictive performance.

Comparing AUCs in R

Comparing multiple AUCs involves calculating the AUC for each model and visualizing the results side by side. This comparison can be done using the pROC package, which provides tools for visualizing and comparing the ROC curves and AUCs of different models. To compare multiple AUC values in R, you generally follow these steps:

Step 1: Install and Load Necessary Packages

First, install and load the required packages. We will use the pROC package for AUC calculations and comparisons, and ggplot2 for visualizations.

install.packages("pROC")
install.packages("ggplot2")
library(pROC)
library(ggplot2)

Step 2: Generate Sample Data

Let’s create some sample data for this demonstration. We will generate three sets of predicted probabilities and their corresponding actual labels.

set.seed(123)
n <- 100  # Number of observations

# Generate actual labels
actual <- rbinom(n, 1, 0.5)

# Generate predicted probabilities for three different models
pred_model1 <- runif(n)
pred_model2 <- runif(n)
pred_model3 <- runif(n)

# Combine data into a data frame
data <- data.frame(
  actual = actual,
  pred_model1 = pred_model1,
  pred_model2 = pred_model2,
  pred_model3 = pred_model3
)

Step 3: Calculate AUC for Each Model

Calculate the AUC for each model using the roc function from the pROC package.

# Calculate AUC for each model
roc_model1 <- roc(data$actual, data$pred_model1)
roc_model2 <- roc(data$actual, data$pred_model2)
roc_model3 <- roc(data$actual, data$pred_model3)

auc_model1 <- auc(roc_model1)
auc_model2 <- auc(roc_model2)
auc_model3 <- auc(roc_model3)

# Print AUC values
cat("AUC for Model 1:", auc_model1, "\n")
cat("AUC for Model 2:", auc_model2, "\n")
cat("AUC for Model 3:", auc_model3, "\n")

Output:

AUC for Model 1: 0.5258932 

AUC for Model 2: 0.5551987 

AUC for Model 3: 0.5873143 

Step 4. Compare AUCs Using DeLong’s Test

To statistically compare the AUCs, we can use DeLong’s test, which is implemented in the roc.test function from the pROC package.

# Compare AUCs using DeLong's test
test_1_vs_2 <- roc.test(roc_model1, roc_model2)
test_1_vs_3 <- roc.test(roc_model1, roc_model3)
test_2_vs_3 <- roc.test(roc_model2, roc_model3)

# Print test results
cat("Comparison of Model 1 vs Model 2:\n")
print(test_1_vs_2)

cat("Comparison of Model 1 vs Model 3:\n")
print(test_1_vs_3)

cat("Comparison of Model 2 vs Model 3:\n")
print(test_2_vs_3)

Output:

Comparison of Model 1 vs Model 2:

    DeLong's test for two correlated ROC curves

data:  roc_model1 and roc_model2
Z = -0.33959, p-value = 0.7342
alternative hypothesis: true difference in AUC is not equal to 0
95 percent confidence interval:
 -0.1984455  0.1398345
sample estimates:
AUC of roc1 AUC of roc2 
  0.5258932   0.5551987 

Comparison of Model 1 vs Model 3:

    DeLong's test for two correlated ROC curves

data:  roc_model1 and roc_model3
Z = -0.73614, p-value = 0.4616
alternative hypothesis: true difference in AUC is not equal to 0
95 percent confidence interval:
 -0.2249540  0.1021118
sample estimates:
AUC of roc1 AUC of roc2 
  0.5258932   0.5873143 

Comparison of Model 2 vs Model 3:

    DeLong's test for two correlated ROC curves

data:  roc_model2 and roc_model3
Z = -0.40243, p-value = 0.6874
alternative hypothesis: true difference in AUC is not equal to 0
95 percent confidence interval:
 -0.1885297  0.1242984
sample estimates:
AUC of roc1 AUC of roc2 
  0.5551987   0.5873143

Step 5. Visualize ROC Curves and AUCs

Use ggplot2 to visualize the ROC curves of all models in a single plot for easier comparison.

# Create a data frame for plotting ROC curves
roc_data <- data.frame(
  specificity = c(roc_model1$specificities, roc_model2$specificities, 
                  roc_model3$specificities),
  sensitivity = c(roc_model1$sensitivities, roc_model2$sensitivities, 
                  roc_model3$sensitivities),
  model = factor(rep(c("Model 1", "Model 2", "Model 3"), 
                     each = length(roc_model1$specificities)))
)

# Plot ROC curves using ggplot2
ggplot(roc_data, aes(x = 1 - specificity, y = sensitivity, color = model)) +
  geom_line(size = 1) +
  labs(title = "ROC Curves for Multiple Models",
       x = "1 - Specificity",
       y = "Sensitivity") +
  theme_minimal()

Output:

Comparing multiple AUCs parallel in R

Conclusion

Comparing multiple AUCs in parallel is an essential task in evaluating the performance of different models. By using the pROC package and ggplot2, you can calculate, compare, and visualize the AUCs of different models effectively. This approach allows you to statistically test the differences between AUCs and provide a clear visual comparison of model performance.