Batch vs real-time inference

In Lesson 3 you put Dev's meal-kit cancellation model behind a live API: the billing system POSTs one customer, the model answers in a fraction of a second. That is one way to serve a model, called real-time, and it is perfect when a system is waiting on a single answer right now.

But the very same model does another job every week that looks nothing like that. Every Monday, marketing wants a list of all the subscribers likely to cancel this week, so they can email them a win-back offer. Nobody is sitting there waiting. The list just needs to be ready by 9am.

Same model, two completely different jobs. This lesson is about the second way to serve a model, batch, and, more importantly, how to look at a decision and know which way fits.

By the end of this lesson you will be able to:

• Explain batch and real-time (online) inference, and run each one in R
• Use three questions, who is waiting, how fresh, and how much it costs, to choose the right pattern for a decision
• Quantify how stale a batch prediction gets, and reach for the precompute-and-cache hybrid when you want both

Prerequisites: Lesson 3 (the live prediction API) and you can [fit a model and read predict output](Your-First-End-to-End-Model-in-R.html).

The same model, two very different jobs

Hold the two jobs side by side. They use identical code to make a prediction, predict(model, customer), and yet almost everything around that one line is different.

The email job scores everyone in advance and stores the answers. The cancel page scores one person the instant they act. Those are the two serving patterns, and the columns above are exactly the questions that will tell you which to use. Let us build each one, then come back to the questions.