Using textrecipes as a sparse matrix engine

library(textrecipes)
library(recipes)

This vignette walks through getting a document-term matrix, such as one to hand to glmnet, xgboost, a clustering routine, or your own code, straight out of a recipe. Build a recipe, prep() it on your training text, then bake() with composition = "dgCMatrix".

Creating a sparse matrix

We will use a handful of short documents.

train <- tibble::tibble(
  text = c(
    "the cat sat on the mat",
    "the dog chased the cat",
    "dogs and cats are pets",
    "a mat is a small rug"
  )
)

new_docs <- tibble::tibble(
  text = c("the cat chased a small dog", "a rug for the mat")
)

Set sparse = "yes" in the step to get sparse output.

rec <- recipe(~text, data = train) |>
  step_tokenize(text) |>
  step_tfidf(text, sparse = "yes") |>
  prep()

Baking the training data with composition = "dgCMatrix" returns a sparse matrix from the Matrix package, one row per document and one column per token.

x_train <- bake(rec, new_data = NULL, composition = "dgCMatrix")
x_train
#> 4 x 16 sparse Matrix of class "dgCMatrix"
#>   [[ suppressing 16 column names 'tfidf_text_a', 'tfidf_text_and', 'tfidf_text_are' ... ]]
#>                                                                           
#> [1,] .         .         .         0.1831020 .         .         .        
#> [2,] .         .         .         0.2197225 .         0.3218876 0.3218876
#> [3,] .         0.3218876 0.3218876 .         0.3218876 .         .        
#> [4,] 0.5364793 .         .         .         .         .         .        
#>                                                                          
#> [1,] .         .         0.183102 0.2682397 .         .         0.2682397
#> [2,] .         .         .        .         .         .         .        
#> [3,] 0.3218876 .         .        .         0.3218876 .         .        
#> [4,] .         0.2682397 0.183102 .         .         0.2682397 .        
#>                         
#> [1,] .         0.3662041
#> [2,] .         0.4394449
#> [3,] .         .        
#> [4,] 0.2682397 .

The prepped recipe remembers the vocabulary and the inverse document frequencies learned from the training data, so new documents are mapped onto the same columns.

x_new <- bake(rec, new_data = new_docs, composition = "dgCMatrix")
x_new
#> 2 x 16 sparse Matrix of class "dgCMatrix"
#>   [[ suppressing 16 column names 'tfidf_text_a', 'tfidf_text_and', 'tfidf_text_are' ... ]]
#>                                                                                
#> [1,] 0.2682397 . . 0.183102 . 0.2682397 0.2682397 . . .         . . .         .
#> [2,] 0.4023595 . . .        . .         .         . . 0.2746531 . . 0.4023595 .
#>                         
#> [1,] 0.2682397 0.1831020
#> [2,] .         0.2746531

identical(colnames(x_train), colnames(x_new))
#> [1] TRUE

Tokens that weren’t seen during prep() are dropped, so the columns always line up between the training and new data.

The result is a regular dgCMatrix, which is the format that many R packages take as input.

# Fit a lasso model with glmnet
fit <- glmnet::glmnet(x_train, y_train, family = "binomial")
predict(fit, newx = x_new)

Steps that produce sparse data

These steps have a sparse argument, which needs to be set to "yes" to get sparse output:

  • step_tf(): term frequencies
  • step_tfidf(): term frequency-inverse document frequency
  • step_texthash(): feature hashing of tokens
  • step_dummy_hash(): feature hashing of nominal variables

Controlling the number of columns

A vocabulary can grow very large, so you will often want to put a cap on the number of columns. There are two options.

Use step_tokenfilter() to keep only the most frequent tokens.

recipe(~text, data = large_corpus) |>
  step_tokenize(text) |>
  step_tokenfilter(text, max_tokens = 5000) |>
  step_tf(text, sparse = "yes") |>
  prep() |>
  bake(new_data = NULL, composition = "dgCMatrix")

Or use feature hashing with step_texthash(). The number of columns is fixed up front with num_terms, no matter how large the vocabulary is, and no vocabulary has to be stored. This makes it a good match for sparse output when the vocabulary is too big to hold in memory.

recipe(~text, data = large_corpus) |>
  step_tokenize(text) |>
  step_texthash(text, num_terms = 4096, sparse = "yes") |>
  prep() |>
  bake(new_data = NULL, composition = "dgCMatrix")

step_dummy_hash() does the same for nominal predictors with many levels.

Using workflows instead

If you are fitting a model with tidymodels, you don’t need to set sparse yourself. A workflow turns it on for models that support sparse data. See ?sparse_data in the recipes package for details.