A step() function is a piecewise constant function that changes its value only at specified points. It is often used to represent discrete data or to create step plots, cumulative distribution functions, or staircase functions.

Step Function in R

The step() function in R Programming Language is used for stepwise variable selection in linear models. It automates the process of selecting a subset of variables from a larger set based on some criterion, such as AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion). Stepwise selection can be forward, backward, or both.

The basic syntax of the step function is:

Syntax:

step(object, direction = c(“both”, “forward”, “backward”), scope, scale = 0)

• object: A fitted model object, typically the result of lm() the function.
• direction: Specifies the direction of stepwise selection.
• scope: A list of components lower and upper.
• scale: A numeric value that controls the step size in forward and backward steps.

Let’s explain the usage with a simple linear regression model.

# Create some sample data
set.seed(123)
x <- rnorm(100)
y <- 2*x + rnorm(100)

# Fit the initial model
initial_model <- lm(y ~ x)

# Perform stepwise variable selection
step_model <- step(initial_model)

# View the summary of the selected model
summary(step_model)

Output:

Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.9073 -0.6835 -0.0875  0.5806  3.2904 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.10280    0.09755  -1.054    0.295    
x            1.94753    0.10688  18.222   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9707 on 98 degrees of freedom
Multiple R-squared:  0.7721,    Adjusted R-squared:  0.7698 
F-statistic:   332 on 1 and 98 DF,  p-value: < 2.2e-16

This tests the overall significance of the model. A low p-value indicates that the model is significant.

• Residuals: These are the differences between the observed values and the predicted values by the model. They measure the model’s accuracy, with smaller residuals indicating a better fit.
• Coefficients: These are the estimates of the intercept and slope of the linear regression line. The intercept represents the value of the dependent variable when the independent variable is zero, while the slope represents the change in the dependent variable for a one-unit change in the independent variable.
• Significance: The significance codes indicate the level of significance of each coefficient. In this case, both the intercept and the slope are highly significant, as denoted by the ‘***’ next to their estimates.
• Residual standard error: This is an estimate of the standard deviation of the residuals. It measures the average deviation of the observed values from the fitted values by the model.
• Multiple R-squared: This is a measure of the proportion of variability in the dependent variable that is explained by the independent variable(s). In this case, approximately 77.21% of the variability in the dependent variable is explained by the independent variable.
• Adjusted R-squared: This is the R-squared value adjusted for the number of predictors in the model. It penalizes the addition of unnecessary predictors.
• F-statistic: This is a test statistic for the overall significance of the model. It compares the fit of the intercept-only model to the fit of the current model. In this case, the F-statistic is 332, with a very low p-value, indicating that the model as a whole is highly significant.

Overall, this summary provides information on the model’s goodness-of-fit, significance of individual predictors, and overall model significance.

Stepwise Selection Options

There are several Stepwise Selection Options are available so we can use them on different scenarios.

1. Forward Selection

Forward selection starts with an empty model and gradually adds predictors one at a time based on their individual contribution to the model fit. The process continues until no additional predictors improve the model fit according to a predetermined criterion.

Syntax:

forward_model <- step(lm(y ~ 1), direction = “forward”)

2. Backward Selection

Backward selection starts with a full model containing all predictor variables and removes one variable at a time until no further removal improves the model fit.

Syntax:

backward_model <- step(initial_model, direction = “backward”)

3. Both Forward and Backward Selection

Both forward and backward selection combines the processes of forward and backward selection. It starts with an empty model and alternates between adding and removing predictor variables until no further improvement can be made.

Syntax:

both_model <- step(initial_model, direction = “both”)

Stepwise selection can lead to overfitting and unstable results, so use it judiciously. Always validate the selected model using techniques like cross-validation.

What if we don’t use the step function?

When you don’t use the step function for variable selection, you’re fitting the model with all available predictors. In the provided output, only one predictor (x) is used. If there were multiple predictors in your dataset, they would all be included in the model unless explicitly excluded. the main difference between fitting the model without using the step function and using it lies in the selection of predictors. The step function facilitates automatic variable selection, potentially leading to a more parsimonious and interpretable model.

Conclusion

The step() function in R provides a convenient way to perform stepwise variable selection in linear models. Understanding its usage and options allows for more informed model building and selection processes. However, it’s essential to interpret the results carefully and consider the potential drawbacks of stepwise selection.