Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 6 of 7

Adversarial Robustness

Lesson 5 watched a model fail a whole group of ordinary customers by accident. Every failure in this course so far, drifting data, a strange input, a sacrificed subgroup, has been unintentional. This lesson introduces the first failure with a mind behind it: an adversary who studies your model and crafts an input designed to fool it.

Meet Leo. He runs the spam filter for a mailing service, and by every honest measure it works: on real inbox traffic it is right about 84% of the time, and it flags one particular junk email as spam with probability 0.81, confident and correct. Then a spammer who has read the same textbook you are about to read makes a change to that email so small it barely registers, and the same filter waves it through as not spam, just as confidently.

Nothing about the model was broken. It is accurate and brittle at the same time, and those two things turn out to be almost unrelated. Toggle the panel below from the original point to a tiny perturbation and watch a confident prediction cross the line.

By the end of this lesson you will be able to:

  • Define an adversarial example and explain how a model can be accurate on normal data yet flip on a tiny crafted nudge
  • See why the attack direction is simply the gradient of the model's score, and build the fast gradient sign method (FGSM) in R
  • Reason about the perturbation budget \(\epsilon\) and how attack success grows with it
  • Harden a model with adversarial training, read the robustness-versus-accuracy tradeoff, and tie robustness to a threat model

Prerequisites: you finished Lesson 5: Group Robustness and DRO, you can fit and read a logistic regression, and you are comfortable with a gradient as "the direction that changes an output fastest."

The core idea

Accurate is not robust

An adversarial example is an input that a human reads one way, that the model should read the same way, but that has been nudged just enough to make the model read it the opposite way. The nudge is deliberate and tiny. The prediction it produces is confident and wrong.

The word to sit with is deliberate. A drifting feature or an out-of-distribution input is the world changing on its own. An adversarial example is someone with access to your model (or a copy of it) solving a small optimization problem: "what is the least I can change about this input to flip the answer?" Accuracy measures how the model does on inputs drawn from nature. Robustness measures how it does on inputs drawn by an opponent. A model can be superb at the first and hopeless at the second.

Each lesson runs in its own R session, so we build Leo's world from scratch. His filter reads two standardized features of every email: link_ratio, how link-heavy the message is (the spammer writes this), and sender_risk, the sending domain's server-side risk score (computed from Leo's logs). Higher means spammier. Real spam depends on both.

RInteractive R
set.seed(3) n <- 1000 link_ratio <- rnorm(n) # how link-heavy the email is (standardized) sender_risk <- rnorm(n) # the sending domain's server-side risk score (standardized) # The TRUE label: genuine spam is driven by BOTH signals. spam <- rbinom(n, 1, plogis(2.2 * link_ratio + 2.2 * sender_risk)) emails <- data.frame(spam, link_ratio, sender_risk) clf <- glm(spam ~ link_ratio + sender_risk, family = binomial, data = emails) round(mean((predict(clf, type = "response") > 0.5) == emails$spam), 2) # accuracy on real email #> [1] 0.84

  

  





The filter is 84% accurate. Now take one ordinary spam email and ask the filter how sure it is:




  

    
RInteractive R

    
email <- data.frame(link_ratio = 0.35, sender_risk = 0.30)     # one junk email
round(as.numeric(predict(clf, email, type = "response")), 3)   # P(spam): confident, and correct
#> [1] 0.811

    

      
      
    

    

  

  





Key Insight
Accuracy and robustness are different properties, not two ends of one scale. Leo's filter is accurate because the boundary it learned separates real spam from real mail well. It is fragile because that same boundary can sit a hair away from a confident point, and an adversary is allowed to aim straight at it.