Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 7

Multi-Armed Bandits, Explore vs Exploit

Lesson 4 ended on an impatient question. Every design in this course so far splits the traffic once and holds it until the planned verdict, even when one arm is clearly pulling ahead by the halfway mark. So why keep sending half your customers to the option you already suspect is losing?

Meera is about to feel that question for real. Her clothing store's Diwali sale opens tomorrow, she has 40,000 loyalty members to email, and three subject lines she cannot choose between. Every email is a one-time shot: a member who ignores a weak subject line is not getting a second, better one later. The campaign IS the experiment, so every send spent proving a loser is an open, a click, a sale, gone for good. This lesson is about spending that traffic wisely, learning which line wins WHILE the campaign is still paying out, and the one number that measures how well you did it: regret.

By the end of this lesson you will be able to:

  • Frame a live campaign as a multi-armed bandit and state the explore-exploit dilemma
  • Define regret, the cumulative cost of learning, and compute it for any allocation
  • Implement epsilon-greedy, the simplest adaptive rule, in a dozen lines of R
  • Explain why its regret climbs in a straight line, and what the best achievable regret is
  • Decide when a bandit beats a fixed A/B split, and when a fixed split is still the right call

Prerequisites: Lessons 1 to 4 of this course (Designing Experiments for Power, the interference designs, and the peeking pitfall) for the fixed-split machinery this lesson breaks from, plus the plain A/B test and short base R loops (Control Flow in R). The reward curves below preview where we are headed.

The setting

A campaign is a row of one-armed bandits

A slot machine has an old nickname: the one-armed bandit. You pull its arm, it pays out or it does not, and it quietly takes your money over time. Picture Meera standing in front of three of them, one per subject line. She does not know any machine's payout rate. She has 40,000 pulls, one per member, and she wants to walk away with as many opens as she can.

That picture is the whole problem, and it comes with a standard vocabulary:

  • An arm is one option she can play. Here the arms are the three subject lines, A, B and C.
  • A pull (or round) is one play of one arm: send that subject line to the next member.
  • The reward is what comes back: 1 if the member opens the email, 0 if not.

Each line has a true open rate, the long-run fraction of members who would open it. Our simulator knows those rates, so we can grade strategies honestly; Meera never sees them. A single send is just one flip of a biased coin:

RInteractive R
set.seed(42) rates <- c(A = 0.30, B = 0.50, C = 0.55) # true open rates, hidden from Meera n_lines <- 3 # One send is a single coin flip: line C is opened with probability 0.55. opens_in_10 <- rbinom(10, 1, rates["C"]) opens_in_10 #> [1] 0 0 1 0 0 1 0 1 0 0 sum(opens_in_10) # opens out of these 10 sends of line C #> [1] 3

  

  





Line C's true rate is 0.55, yet these particular ten sends opened only 3 times. That gap between the true rate and what you actually see is the entire difficulty: with a handful of sends, a good line can look bad and a bad line can look good. Meera has to decide where to send email after email while staring at exactly this kind of noisy tally.