Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 6 of 7

Thompson Sampling and Bayesian Bandits

Lesson 5 convicted epsilon-greedy on two counts. Its exploration is blind: a random email is as likely to go to hopeless subject line A as to the genuinely open B-versus-C question. And it never stops: one send in ten stays random forever, a straight-line tax long after the answer is settled.

Both flaws share a root. Epsilon-greedy summarizes each subject line with a single number, its observed open rate, and a single number cannot say how sure it is. This lesson gives Meera, 60 sends into her 40,000-email Diwali campaign, the fix: keep a whole curve of belief per line and let the curves run the campaign. The rule that falls out is almost embarrassingly simple, one random draw per line per send, yet it allocates traffic exactly in proportion to the probability each line is currently best, explores only where real uncertainty remains, and its regret provably bends toward the Lai-Robbins floor that epsilon-greedy misses. It is called Thompson sampling, it predates the computer, and it is the engine inside most modern experimentation platforms.

By the end of this lesson you will be able to:

  • Turn each line's tally into a Beta belief curve and update it after every send
  • Compute the probability each line is truly best by simulating from those curves
  • Implement Thompson sampling in a dozen lines of base R and read its allocation
  • Explain why its exploration fades on its own and its regret curve bends where epsilon-greedy's climbs
  • Say what it does not fix: biased tallies, delayed rewards, and drifting rates

Prerequisites: Multi-Armed Bandits, Explore vs Exploit, this lesson picks up its campaign mid-send: arms, rewards, cumulative regret and epsilon-greedy are assumed. The Beta-posterior machinery comes from the Bayesian course (The Bayesian Update, Conjugacy and Choosing Priors); a working refresher is built in, so a comfortable memory of it is enough.

The belief

A curve of belief per subject line

Rejoin Meera exactly where the Lesson 5 dilemma table left her: 60 of the 40,000 Diwali emails sent, and a scoreboard. Line A, "The Diwali sale is on", opened 4 of 15. Line B, "40 percent off ends Friday", opened 11 of 20. Line C, "Your size is going fast", opened 12 of 25.

Epsilon-greedy reduces each row to one number: 0.27, 0.55, 0.48. The Bayesian course taught a richer bookkeeping: treat line \(k\)'s true open rate \(p_k\) as an unknown and keep a whole probability distribution over what it might be. Start every line at Beta(1, 1), the flat prior that calls every rate from 0 to 1 equally plausible, and let conjugacy absorb each result as it arrives. After \(s_k\) opens in \(n_k\) sends, the belief about line \(k\) is

\[ p_k \mid \text{data} \;\sim\; \text{Beta}(s_k + 1,\; n_k - s_k + 1) \]

Every symbol in words: \(s_k\) is line \(k\)'s opens so far and \(n_k\) its sends, so the first shape parameter counts opens plus the prior's one, and the second counts ignores (sends minus opens) plus one. The curve's center of mass sits at \((s_k + 1)/(n_k + 2)\), essentially the observed rate, and its width is the honesty that a lone number throws away: fewer sends, wider curve.

RInteractive R
rates <- c(A = 0.30, B = 0.50, C = 0.55) # the truth: hidden from Meera, known to our simulator opens <- c(A = 4, B = 11, C = 12) # the 60-send scoreboard from Lesson 5 sends <- c(A = 15, B = 20, C = 25) shape1 <- opens + 1 # opens, plus one from the flat prior shape2 <- sends - opens + 1 # ignores, plus one round(shape1 / (shape1 + shape2), 2) # each curve's center of mass #> A B C #> 0.29 0.55 0.48

  

  





The centers barely differ from the raw rates. The curves are where the new information lives, so draw all three:




  

    
RInteractive R

    
curve(dbeta(x, shape1["C"], shape2["C"]), from = 0, to = 1, n = 400, lwd = 2.5,
      col = "#1b7f5c", xlab = "plausible true open rate", ylab = "plausibility (density)",
      main = "What Meera believes after 60 sends")
curve(dbeta(x, shape1["B"], shape2["B"]), col = "#c25e00", lwd = 2.5, add = TRUE)
curve(dbeta(x, shape1["A"], shape2["A"]), col = "#5b6ee1", lwd = 2.5, add = TRUE)
legend("topright", legend = c("A: 4 opens of 15", "B: 11 opens of 20", "C: 12 opens of 25"),
       col = c("#5b6ee1", "#c25e00", "#1b7f5c"), lwd = 2.5, bty = "n")

    

      
      
    

    

  

  





Read the picture. A's curve sits low and away from the others: 15 sends were enough to make "A is secretly best" a fringe possibility. B's curve peaks to the right of C's, that is the 0.55 versus 0.48 scoreboard talking, but both curves are wide and overlap heavily, so "C is really the best line" remains entirely plausible. The scoreboard said B leads; the curves say the B-versus-C question is still open.



Key Insight
A tally is a number; a belief is a curve. The curve carries the one thing the number drops, how much evidence backs it, and that is exactly the quantity intelligent exploration needs.