dials mtry() in R: Tune Random Forest Variable Sampling

The dials mtry() function in R defines the integer hyperparameter for the number of predictors randomly sampled at each split in tree ensembles. It returns a parameter object with an unknown upper bound, so you almost always pair it with finalize() to fill in the maximum from your training data before tuning.

mtry()                                   # default range 1L to unknown()
mtry(range = c(1L, 20L))                 # explicit integer range
finalize(mtry(), train_data)             # set upper bound from data
update(params, mtry = mtry(c(2L, 8L)))   # override range in a param set
mtry_prop()                              # proportion variant, no finalize needed
mtry_long()                              # log-scaled variant for wide data
grid_regular(mtry(c(1L, 10L)), levels = 5) # candidate grid

What mtry() does in one sentence

mtry() returns a dials parameter object, not a numeric value. It describes the hyperparameter that controls how many predictors are sampled when each split is considered in a random forest or boosted tree. The object carries a range, a label, a transformation (none, by default), and an integer flag, which is enough metadata for grid_regular(), grid_space_filling(), and tune_grid() to build candidate sets and rank them.

The wrinkle that sets mtry() apart from other dials parameters is the upper bound. Most parameters ship with sensible defaults baked in. mtry()'s default is c(1L, unknown()) because the maximum sensible value equals the number of columns in your predictor matrix, which dials cannot know until you hand it data. That is why finalize() exists, and it is the call you forget at your peril.

mtry() syntax and arguments

The signature is intentionally minimal.

The return is a quant_param S3 object. Print it to inspect the range, call value_seq() to generate values, or pass it to grid_*() helpers.

Examples by use case

Start with a tunable random forest, then finalize the range, then grid the candidates.

Extract the parameter set, see the unknown upper bound, then finalize against the training data.

The nparam[?] flips to nparam[+] once the upper bound is set. Now a regular grid actually has both endpoints.

You can skip the finalize step entirely by switching to mtry_prop(), which lives on [0, 1] and does not depend on data shape.

mtry() versus mtry_prop() and mtry_long()

Pick by how you reason about the model, not by what the documentation lists first.

ranger and xgboost both accept the raw integer behind the scenes; the dials variant just controls how the search space is sampled. A 100-predictor dataset with mtry_long() and 5 levels samples roughly at 1, 4, 10, 32, 100, which spreads coverage on the log scale.

Common pitfalls

Four mistakes account for most failed tune_grid() calls involving mtry.

1. Forgetting finalize(). grid_regular(params) with an unfinalized mtry returns Error: Some parameters require finalization. Always pipe through finalize(train_predictors) before constructing the grid.
2. Passing the full data frame including the outcome. finalize() counts columns. A data frame with the outcome inflates the upper bound by one, which is harmless for trees but confusing in value_seq(). Pass train |> select(-Sale_Price) or a recipe-prepped matrix for clarity.
3. Mixing mtry() and mtry_prop() in one parameter set. Only one can occupy the mtry slot of a parsnip spec. If you update from one to the other, the entire mtry parameter object is replaced, not merged.
4. Treating mtry as numeric in a custom tibble grid. A grid with mtry = c(0.5, 0.7) will be coerced to integer 0 and 0, silently breaking the run. Use mtry_prop() for fractions or stick to integer values for mtry().

Try it yourself

Related tidymodels functions

mtry() rarely flies solo; the typical call lives inside a short pipeline.

• finalize() to substitute the unknown upper bound with the training data column count.
• mtry_prop() and mtry_long() for proportion-based and log-scaled variants of the same knob.
• extract_parameter_set_dials() to pull every tunable parameter from a workflow at once.
• update() to override one parameter range inside a parameter set without rebuilding the spec.
• grid_regular(), grid_random(), grid_space_filling() to turn the parameter set into candidate tibbles.
• tune_grid() to actually fit each candidate across resamples.

External reference: the official dials documentation at dials.tidymodels.org.

FAQ

What is mtry in a random forest?

mtry is the number of predictor variables randomly sampled as candidates at each split point during tree construction. It is the single most important hyperparameter for random forests because it controls the correlation between trees in the ensemble. Lower mtry produces more diverse trees and stronger variance reduction; higher mtry produces stronger individual trees but more correlation. The classic defaults are sqrt(p) for classification and p/3 for regression, where p is the number of predictors.

Why does mtry() have an unknown upper bound by default?

Because dials parameter objects are defined before the data is seen. The maximum sensible value for mtry is the number of predictor columns, which depends on the data and any recipe steps that change the column count (step_dummy(), step_pca(), step_zv()). Leaving the upper bound as unknown() forces an explicit finalize() call, which makes the dependency on data shape visible in the code rather than hidden behind a constant.

How is mtry() different from mtry_prop()?

mtry() lives on the integer range [1, p] and tunes the absolute count of sampled predictors. mtry_prop() lives on the numeric range [0, 1] and tunes the proportion. Internally, ranger or xgboost still receives an integer count at fit time, computed as ceiling(mtry_prop * p). Use mtry_prop() when you want the search space to be portable across datasets of different widths, or when you find proportions easier to reason about than counts.

Do I need to call finalize() if I pass an explicit range to mtry()?

No. mtry(range = c(2L, 10L)) is a fully specified parameter object; is_unknown() returns FALSE on both endpoints, and grid_regular() can sample it directly. Use this form when you want to bound the search tighter than the data permits, for example to prevent the model from sampling all predictors and degenerating into a bagged tree ensemble.

Can I tune mtry alongside recipe steps in the same workflow?

Yes. Mark the recipe step with tune() (for example, step_pca(num_comp = tune())) and rand_forest(mtry = tune()) in the model spec. extract_parameter_set_dials(wf) |> finalize(train) collects both. Finalize after adding the recipe to the workflow so the mtry upper bound reflects the post-recipe column count, not the raw frame.