Reproducible pipelines with targets
You can build a model. Now you have to ship it, and keep it trustworthy. Meet Dev, a data scientist at a meal-kit company. He has one script, analysis.R, that predicts which customers will cancel: it reads the raw orders, aggregates them into per-customer numbers, fits a model, and writes a summary table. Start to finish it takes about 40 minutes, because fitting the model is slow.
This afternoon Dev changed the summary table six times, each time to round a number differently. Six times he waited 40 minutes. This lesson is about the tool that ends that: targets, which reruns only the steps that actually changed.
By the end of this lesson you will be able to:
- Explain why one long analysis script is slow to iterate on and lets your results quietly drift out of sync with your code
- Describe a targets pipeline as steps wired into a dependency graph, where each step is the output of a function
- Predict exactly which steps rerun after a change, so a one-character edit costs seconds instead of 40 minutes
Prerequisites: you can fit a model end to end, you can write a function, and it helps to have met reproducibility with renv and git. The four boxes below are Dev's whole analysis.
One script reruns everything
Dev's analysis lives in a single file. To run it, he calls source("analysis.R"), and it executes top to bottom:
# analysis.R - the whole analysis in one file
library(readr); library(dplyr)
orders <- read_csv("orders.csv") # 12,000 raw order rows
features <- get_features(orders) # aggregate to 800 customers
model <- fit_model(features) # trains the model ... ~38 minutes
summary <- summarize_model(model) # the little table at the end
write_csv(summary, "summary.csv")
The trouble is that source() is all-or-nothing. Change the last line, the one that rounds a number in the summary, and R still reruns the 38-minute model fit above it, because it has no idea the model did not change. Every tiny edit costs the full 40 minutes.
So Dev does what everyone does under time pressure: he comments out the slow line and reuses the model object still sitting in his workspace from the last run. It is faster, but now something dangerous has happened. The summary.csv he ships was built from an old model, while the code on disk describes a new one. The results and the code have silently drifted apart, and nobody can tell.
Break the analysis into steps
targets asks you to do one thing first: write each step of the analysis as its own small function, so each step has a clear input and a clear output. Nothing exotic, just ordinary R functions.
First the raw data. Each lesson runs in a fresh R session, so we build a small stand-in for Dev's orders.csv right here. These 200 rows are one per customer: how many boxes they ordered, their total spend, weeks since their last order, and whether they cancelled.
Now each step of the analysis becomes a function, one per box from the cover. One honest simplification: because our stand-in already has one row per customer, get_features here just derives a new column (spend per box) rather than doing Dev's full order-to-customer roll-up. The pipeline's shape is identical either way, and that shape is the whole point:
get_features needs orders, fit_model needs features, summarize_model needs model. Each step consumes the output of the step before it. That chain of "needs" is the thing targets is about to exploit.A pipeline is a dependency graph
Write down that chain of "needs" and you get a dependency graph: a picture of which step depends on which. Dev's analysis is a simple chain. features depends on orders; model depends on features; summary depends on model.
The word to hold onto is downstream. A step is downstream of another if it depends on it, directly or through a chain. model is downstream of orders (through features); summary is downstream of everything. Upstream is the reverse: orders is upstream of all three.
Why does this matter? Because the graph tells targets exactly what a change can and cannot affect. If you edit a step, the only results that can possibly change are that step and the ones downstream of it. Everything upstream is provably safe to reuse. That single fact is the whole trick, and the next step makes it concrete.
What is downstream of features?
Look at the graph: orders -> features -> model -> summary. You edit the features step. Which results might now be wrong and therefore need rebuilding?
model and summary are downstream of features, so a change there can ripple through both. orders is upstream, so it is untouched.orders is upstream of features; nothing you do to a later step can change the raw data that fed it. Only the changed step and its downstream rebuild.Rerun only what changed
Here is how targets decides what to skip. For every step it stores a recipe: the step's code plus the inputs it depends on. Before running a step, it compares the current recipe to the stored one. If they are identical, the result cannot have changed, so it reuses the saved result and moves on. If the recipe differs, it rebuilds that step, and because the rebuilt output is a new input, every step downstream rebuilds too.
That is the entire idea, and it fits in about ten lines you can run. Here is a toy version of it:
Now the three steps. Each step's recipe is its code plus the value of the step it depends on, so a rebuilt upstream automatically changes a downstream recipe:
Now the two cases that matter. Edit only the summary, and only the summary rebuilds. Change the raw data, and the change cascades through everything downstream:
Wire a step to its input
A downstream step is defined by what it consumes. You have get_features(), fit_model() and summarize_model() loaded from earlier, plus the features object you built. The model step should be fed the features. Fill in the blank so the model trains on the right input.
Show answer
model2 <- fit_model(features)Put the pipeline in _targets.R
You express all of this in one special file named _targets.R. It loads your functions and returns a list of targets: one tar_target(name, recipe) per step. The recipe just calls your function, and it declares a dependency simply by naming another target inside it, exactly the fit_model(features) wiring you just wrote.
# _targets.R - the pipeline definition
library(targets)
tar_source() # loads get_features(), fit_model(), summarize_model() from the R/ folder
list(
tar_target(orders_file, "orders.csv", format = "file"), # track the raw file itself
tar_target(orders, read_csv(orders_file)),
tar_target(features, get_features(orders)),
tar_target(model, fit_model(features)), # the slow step
tar_target(summary, summarize_model(model))
)
Then you drive the pipeline with three commands from the R console:
tar_make() # build every target that is out of date, skip the rest
tar_visnetwork() # see the dependency graph, colored by what is up to date
tar_read(summary) # pull any target's result back into R to inspect it
format = "file" is the piece that watches your data. It tells targets to hash orders.csv itself, so if the raw file changes, everything downstream rebuilds; if it does not, nothing does. targets tracks the dependencies you declare (target-to-target links and files); it does not pin package versions or capture your operating system. That layer is renv and, for full isolation, a container.Change one thing, rebuild one thing
Now replay Dev's afternoon with the pipeline in place. He edits summarize_model() to round to one decimal and runs tar_make() again. targets checks each target's recipe against what it stored last time. Only the summary's recipe changed, so only the summary rebuilds. The 38-minute model is untouched.
The console log makes the skipping visible. The exact wording and symbols vary by targets version, but it reads roughly like this:
tar_make()
#> skip orders_file
#> skip orders
#> skip features
#> skip model
#> run summary
#> completed summary [0.5 seconds]
Six edits to the summary now cost about three seconds total instead of four hours. And the stale-object temptation is gone: because targets always reuses the correct cached result and rebuilds anything genuinely out of date, the output you ship is guaranteed to match the current code. Fast and trustworthy, at the same time.
What actually reruns?
Dev's pipeline is built and up to date. He edits only summarize_model() to change the rounding, saves, and runs tar_make(). The model fit takes 38 minutes. What happens?
References
A few authoritative, free places to take this further:
- The {targets} R package user manual - the canonical, book-length guide to pipelines, dependencies and caching.
- A walkthrough to get started - build your first working pipeline from an empty folder, step by step.
- tar_make() reference - the exact behavior of the command that builds only out-of-date targets.
- Landau (2021), The targets R package, JOSS - the short peer-reviewed paper describing the design.
Lesson 1 complete
You turned a fragile 40-minute script into a pipeline that reruns only what changed. The moves: write each step as a function, let their inputs and outputs form a dependency graph, and let targets skip any step whose recipe (code plus inputs) is unchanged while rebuilding the changed step and everything downstream. The result is an analysis that is both fast to iterate on and guaranteed to stay in sync with its code.
Next, Lesson 2: Versioning models with vetiver and pins. Your pipeline produces a trained model; now you will register, version and retrieve that model the same disciplined way you version code, so you always know exactly which model is in production.