Target Encoding Without Leakage

In Lesson 1, Maya turned her used-car listings into numbers a model can read: one-hot for body, ordinal for condition. One column fought back. brand has dozens of makes, and one-hot exploded it into dozens of sparse columns that break on any make her data had never seen.

We ended with a tempting fix: replace each brand with the average price of cars in that brand. One tidy numeric column, and a strong one, because expensive makes really do sell for more. It is called target encoding, and it is the most powerful way to tame a high-cardinality column. It is also the easiest to get catastrophically wrong.

By the end of this lesson you will be able to:

• Compute a target encoding: replace a category with the average outcome of its rows
• See, in real numbers, how a careless target encoding leaks the answer and makes even pure noise look predictive
• Encode out-of-fold so no row ever sees its own outcome, and smooth rare categories so they behave
• Encode brand-new data the model has never seen, safely

Prerequisites: you can run R and read its output, you met encoding in Encoding Categorical Variables, and you know what a train/test split and data leakage are (from Train, Validation, Test, and Data Leakage).

Here is the whole journey. We earn each step.

A category becomes its average outcome

Target encoding answers the high-cardinality problem with one move: replace each category with the average of the target over the rows in that category. For Maya, the target is price, so each brand becomes the mean price of the cars that carry it. Toyotas that averaged $14k become 14; BMWs that averaged $27k become 27. The dozens of brand names collapse into a single, meaningful number.

Written out, the target encoding of a category \(c\) is

\[ \text{TE}(c) \;=\; \frac{1}{n_c} \sum_{i \,:\, \text{brand}_i = c} y_i, \]

where \(y_i\) is the target for row \(i\) (here, the car's price), \(n_c\) is the number of rows whose brand equals \(c\), and the sum runs over exactly those rows. In words: add up the prices of every car of that brand and divide by how many there are. That is all target encoding is, an average.

Notice one thing before we go on, because it turns out to be the whole story of this lesson: for a brand with many cars the average is stable, but for a brand with a single car the "average" is just that one car's price. Watch the code column appear.