Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 2 of 6

Permutation and Drop-Column Importance

In Lesson 1 you read a global feature-importance ranking off a churn model, and you did it the easy way: the model was a logistic regression, so each feature had a coefficient, and the size of the coefficient told you how much the feature mattered.

Then your team swapped in a random forest, because it predicts churn better. It has no coefficients. Hundreds of trees vote; there is no single number per feature to read. So the old recipe is dead, and yet the product lead still wants the same answer: which features drive this model?

This lesson answers that for any model, even a black box, with two model-agnostic methods, and then shows the honest ways they can quietly mislead you.

By the end you will be able to:

  • Compute permutation importance in R: shuffle one feature and measure how far accuracy falls
  • Compute drop-column importance in R: retrain without a feature and measure the same fall, and say when each method is worth its cost
  • Spot the trap that fools both: why a genuinely important feature can look worthless when a correlated twin is present

Prerequisites: you can fit and use a model in R such as a random forest, you know what a train/test split and accuracy are, and you have done Lesson 1: Global vs Local Explanations.

The problem

A model with no coefficients

Here is the same churn data from Lesson 1, but this time we fit a random forest instead of a logistic regression. Each row is a customer; churned is "yes" if they left. Build it and fit once.

RInteractive R
library(randomForest) set.seed(42) n <- 500 churn <- data.frame( tenure = round(runif(n, 0, 60)), # months as a customer monthly = round(runif(n, 20, 120), 1), # monthly charge (dollars) support_calls = rpois(n, 1.5), # support calls last quarter contract = rbinom(n, 1, 0.5), # 1 = on a 1-year contract addons = rbinom(n, 1, 0.4), # has paid add-ons senior = rbinom(n, 1, 0.16) # senior-citizen flag ) # who leaves depends mostly on short tenure, high charges, and support calls lp <- -1.0 - 0.06 * churn$tenure + 0.02 * churn$monthly + 0.35 * churn$support_calls - 0.5 * churn$contract churn$churned <- factor(ifelse(rbinom(n, 1, plogis(lp)) == 1, "yes", "no")) set.seed(1) i <- sample(nrow(churn), 350) train <- churn[i, ] # fit the model here test <- churn[-i, ] # judge it here rf <- randomForest(churned ~ ., data = train, ntree = 120) base_acc <- mean(predict(rf, test) == test$churned) # accuracy on unseen customers round(base_acc, 3) #> [1] 0.789

  

  





The forest predicts the held-out customers correctly about 79% of the time. Good enough. But coef(rf) does not exist, and there is no \(\beta_j\) to scale. To rank features for a model like this, we cannot look inside it. We have to poke it from the outside and watch how it reacts. That is what "model-agnostic" means: a method that only needs to feed the model inputs and read its outputs, so it works on a forest, a neural net, or anything else.