Contextual Bandits and Off-Policy Evaluation
Lesson 6 ended the Diwali campaign on a high: Thompson sampling routed Meera's 40,000 emails at half the regret of epsilon-greedy, and faded its own exploration with no knob to tune. But one assumption survived all six lessons unexamined: that a single subject line is best for everyone. Meera's list says otherwise. "Your size is going fast" lands with longtime loyalty members; "40 percent off ends Friday" lands with bargain hunters. A bandit that could see who it is emailing would beat any one-size-fits-all winner.
This finale does two jobs. First it teaches the bandit to see: a contextual bandit lets the best line depend on the reader, and the upgrade costs a dozen lines of base R. Then it answers the question every finished campaign begs: Meera now has a smarter policy for next year, so can she score it on this year's log, without sending a single new email? That is off-policy evaluation, and its two workhorses, inverse-propensity scoring and the doubly robust estimator, close the course.
By the end of this lesson you will be able to:
- Explain what a context is, and what a context-blind bandit pays, forever, for not seeing it
- Upgrade Thompson sampling to a per-segment contextual bandit in base R and read its allocation
- Read a bandit log, say what a propensity is, and show why the log's naive averages mislead
- Score a brand-new policy from logged data alone with inverse-propensity scoring
- Strengthen that score with the doubly robust estimator, and say when no estimator can help
Prerequisites: Thompson Sampling and Bayesian Bandits, this lesson starts the morning after that campaign ends: Beta belief curves, probability-of-best, and regret are assumed, as is the campaign itself from Multi-Armed Bandits, Explore vs Exploit. All code is base R.
The best line depends on who is reading
Two days after the campaign closes, Meera's analyst slices the final tallies by a column the bandit never looked at: the membership segment. 24,000 of the 40,000 subscribers are bargain hunters, signed up through discount partners, and 16,000 are loyalty members, two or more years of full-price orders. Split that way, the scoreboard stops telling one story and starts telling two: line B ("40 percent off ends Friday") clearly led among bargain hunters, while line C ("Your size is going fast") clearly led among loyalty members.
As always in this course, the simulator knows the truth that Meera can only estimate. Here it is, one open rate per segment and line, and the blend checks out against the campaign-wide rates the whole course has used:
C's famous 0.55 was never one number. It was a 0.49 among bargain hunters and a 0.64 among loyalty members, averaged by the list mix. Draw the two segments side by side and the flip is impossible to miss:
The segment is a context: a fact Meera knows about each recipient before choosing which line to send. A rule that looks at the context and names a line, "B to bargain hunters, C to loyalty members", is called a policy. A policy \(\pi\) (the Greek letter pi, the usual symbol for a policy) has a value: the campaign-wide open rate it would earn,
\[ V(\pi) \;=\; \sum_{x} \Pr(x)\; p_{x,\,\pi(x)} \]
where \(x\) runs over the segments, \(\Pr(x)\) is the share of the list in segment \(x\) (0.60 and 0.40 here), \(\pi(x)\) is the line the policy picks for that segment, and \(p_{x,a}\) is the true open rate of line \(a\) in segment \(x\). Compare the best any single line can do with the best per-segment rule: