Encoding Categorical Variables

Maya runs a small used-car listing site. She wants to predict a car's resale price from its details, and her data describes each car with words: transmission is manual or automatic, the body is sedan, hatchback, or SUV, the condition runs poor to excellent, and the brand is one of dozens of makes. A model, though, only does arithmetic. It cannot multiply the word "automatic" by anything.

Encoding is how we hand a model those words as numbers, and doing it carelessly teaches the model things that are not true. By the end of this lesson you will be able to:

• Explain why numbering categories 1, 2, 3 can mislead a model
• One-hot, dummy, and ordinal encode variables in R, and read the result
• Pick an encoding for a column with hundreds of categories, and say what it costs

Prerequisites: you can run R and read its output, and you know what a model, a feature, and a train/test split are (from Train, Validation, Test, and Data Leakage).

There is no single right encoding, only the right one for the kind of category. Here is the whole map; we spend the lesson earning it.

A model only multiplies numbers

Picture the simplest price model Maya could fit, a straight-line one. It predicts a car's price by multiplying each feature by a learned weight and adding them up:

\[ \hat{y} \;=\; \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p. \]

Here \(\hat{y}\) is the predicted price, each \(x_j\) is one feature (a number), each \(\beta_j\) is the weight the model learns for it, and \(\beta_0\) is the intercept, a baseline constant that belongs to no feature. The whole machine is multiplication and addition. So every \(x_j\) has to be a number. The word "automatic" has no value to multiply, and the same is true of every tree, boosting, or neural model underneath: somewhere they all reduce a row to numbers.

Each lesson runs in a fresh R session, so let us build Maya's listings right here and look at what we are dealing with.

Four of the five columns are text. Encoding is the job of turning those four into numbers, and the next steps are about doing it without lying to the model.