Tokenization is the first step in almost every textrecipes workflow. It splits a character column into smaller pieces, called tokens, that later steps can filter, modify, and turn into numeric features. This vignette gives an overview of the tokenization options in the package and how each choice changes the tokens you get.
We will use two short texts throughout.
text_tibble <- tibble::tibble(
text = c(
"This is a sentence. It has two!",
"They're e-mailing Dr. Smith about the 3 cats."
)
)step_tokenize() converts a character column into a token
column (see ?tokenlist). The token argument
picks the unit of tokenization and the engine argument
picks the package doing the work. With no arguments you get word tokens
from the tokenizers package.
recipe(~text, data = text_tibble) |>
step_tokenize(text) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "this" "is" "a" "sentence" "it" "has" "two"
#>
#> [[2]]
#> [1] "they're" "e" "mailing" "dr" "smith" "about" "the"
#> [8] "3" "cats"show_tokens() is a convenient way to see what a recipe
produces. Each element of the result is the token vector for one
row.
The engine determines which values of token
are available.
| Engine | Tokens | Trained on data |
|---|---|---|
tokenizers |
"words", "characters",
"character_shingle", "ngrams",
"skip_ngrams", "sentences",
"lines", "paragraphs", "regex",
"ptb", "word_stems" |
No |
spacyr |
"words" |
No |
tokenizers.bpe |
"words" |
Yes |
udpipe |
"words" |
No (needs a pre-trained model) |
Subword tokenizers such as wordpiece, sentencepiece, and byte pair
encoding also have dedicated steps:
step_tokenize_wordpiece(),
step_tokenize_sentencepiece(), and
step_tokenize_bpe().
The tokenizers engine is the default. Every token option
corresponds to a function in the tokenizers package, and the
options argument takes a named list of arguments passed to
that function.
Word tokenization lowercases the text and removes punctuation by default.
recipe(~text, data = text_tibble) |>
step_tokenize(text, token = "words") |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "this" "is" "a" "sentence" "it" "has" "two"
#>
#> [[2]]
#> [1] "they're" "e" "mailing" "dr" "smith" "about" "the"
#> [8] "3" "cats"Both behaviors are controlled through options, see
?tokenizers::tokenize_words. Here we keep the original case
and the punctuation.
recipe(~text, data = text_tibble) |>
step_tokenize(
text,
options = list(lowercase = FALSE, strip_punct = FALSE)
) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "This" "is" "a" "sentence" "." "It" "has"
#> [8] "two" "!"
#>
#> [[2]]
#> [1] "They're" "e" "-" "mailing" "Dr" "." "Smith"
#> [8] "about" "the" "3" "cats" "."Note that "e-mailing" is split at the hyphen into
"e" and "mailing" by default. Numbers can be
removed by setting strip_numeric = TRUE.
The "ptb" tokenizer follows the Penn Treebank
conventions. It splits contractions such as "They're" into
"They" and "'re" and keeps hyphenated words
intact, which is often what you want for linguistic work.
"characters" returns single characters, and
"character_shingle" returns overlapping character n-grams
whose size is set with n.
recipe(~text, data = text_tibble) |>
step_tokenize(text, token = "characters") |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "t" "h" "i" "s" "i" "s" "a" "s" "e" "n" "t" "e" "n" "c" "e" "i" "t" "h" "a"
#> [20] "s" "t" "w" "o"
#>
#> [[2]]
#> [1] "t" "h" "e" "y" "r" "e" "e" "m" "a" "i" "l" "i" "n" "g" "d" "r" "s" "m" "i"
#> [20] "t" "h" "a" "b" "o" "u" "t" "t" "h" "e" "3" "c" "a" "t" "s"recipe(~text, data = text_tibble) |>
step_tokenize(
text,
token = "character_shingle",
options = list(n = 3)
) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "thi" "his" "isi" "sis" "isa" "sas" "ase" "sen" "ent" "nte" "ten" "enc"
#> [13] "nce" "cei" "eit" "ith" "tha" "has" "ast" "stw" "two"
#>
#> [[2]]
#> [1] "the" "hey" "eyr" "yre" "ree" "eem" "ema" "mai" "ail" "ili" "lin" "ing"
#> [13] "ngd" "gdr" "drs" "rsm" "smi" "mit" "ith" "tha" "hab" "abo" "bou" "out"
#> [25] "utt" "tth" "the" "he3" "e3c" "3ca" "cat" "ats"Character-level tokens are useful for short strings with typos or without clear word boundaries, but they are rarely appropriate for long documents.
"ngrams" returns word n-grams. Use n and
n_min to choose the range of sizes.
recipe(~text, data = text_tibble) |>
step_tokenize(text, token = "ngrams", options = list(n = 2, n_min = 1)) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "this" "this is" "is" "is a" "a"
#> [6] "a sentence" "sentence" "sentence it" "it" "it has"
#> [11] "has" "has two" "two"
#>
#> [[2]]
#> [1] "they're" "they're e" "e" "e mailing" "mailing"
#> [6] "mailing dr" "dr" "dr smith" "smith" "smith about"
#> [11] "about" "about the" "the" "the 3" "3"
#> [16] "3 cats" "cats""skip_ngrams" allows gaps between the words, controlled
by k.
recipe(~text, data = text_tibble) |>
step_tokenize(
text,
token = "skip_ngrams",
options = list(n = 2, n_min = 2, k = 1)
) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "this is" "this a" "is a" "is sentence" "a sentence"
#> [6] "a it" "sentence it" "sentence has" "it has" "it two"
#> [11] "has two"
#>
#> [[2]]
#> [1] "they're e" "they're mailing" "e mailing" "e dr"
#> [5] "mailing dr" "mailing smith" "dr smith" "dr about"
#> [9] "smith about" "smith the" "about the" "about 3"
#> [13] "the 3" "the cats" "3 cats"If you want n-grams of tokens you have already modified, for example
after removing stop words, tokenize to words first and use
step_ngram() instead. See
vignette("Working-with-n-grams").
These tokenizers split at larger boundaries. They are mostly useful as a first step before splitting further, or when each sentence or paragraph is the unit you want to model.
recipe(~text, data = text_tibble) |>
step_tokenize(text, token = "sentences") |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "This is a sentence." "It has two!"
#>
#> [[2]]
#> [1] "They're e-mailing Dr." "Smith about the 3 cats.""lines" splits on line breaks and
"paragraphs" on blank lines.
"regex" splits the text wherever a pattern matches. The
pattern is given with pattern, and the default is a single
space.
"word_stems" tokenizes into words and stems each of them
in one go. If you want to control the stemmer, or stem after other
filtering, tokenize to words and use step_stem().
Some tokenizers learn their vocabulary from the data. Their training
arguments are passed in training_options, while
options is used when the tokenizer is applied. The
tokenizers.bpe engine learns a byte pair encoding on the training set,
and vocab_size is the most important training option. It is
usually in the thousands, but is kept small here to fit our tiny
data.
recipe(~text, data = text_tibble) |>
step_tokenize(
text,
engine = "tokenizers.bpe",
training_options = list(vocab_size = 40)
) |>
show_tokens(text, n = 2)Because the vocabulary is learned during prep(), the
same tokenizer is applied to new data when you bake(). The
dedicated steps step_tokenize_bpe(),
step_tokenize_sentencepiece(), and
step_tokenize_wordpiece() give you the same kind of subword
tokens with arguments specific to each method.
The spacyr engine tokenizes with spaCy and requires a working Python
installation with spaCy. Its options are applied each time the data is
tokenized, including at bake() time.
recipe(~text, data = text_tibble) |>
step_tokenize(text, engine = "spacyr") |>
show_tokens(text, n = 2)The udpipe engine requires a pre-trained model, loaded with
udpipe::udpipe_load_model(), passed as model
in training_options.
When none of the engines do what you need, pass your own function to
custom_token. It must take a character vector and return a
list of character vectors with one element per input.
space_tokenizer <- function(x) {
strsplit(x, " +")
}
recipe(~text, data = text_tibble) |>
step_tokenize(text, custom_token = space_tokenizer) |>
show_tokens(text, n = 2)
#> [[1]]
#> [1] "This" "is" "a" "sentence." "It" "has"
#> [7] "two!"
#>
#> [[2]]
#> [1] "They're" "e-mailing" "Dr." "Smith" "about" "the"
#> [7] "3" "cats."There is rarely a single correct choice, and the best tokenizer depends on your data and model. Some rules of thumb:
"words". It is fast, easy to interpret, and
works well for many bag-of-words models."ptb" or the spacyr and udpipe engines when
linguistic detail such as contractions or part of speech matters.Since the tokenizer is a recipe argument, token can also
be tuned with the tunable method for step_tokenize().
After tokenizing you will typically filter and modify the tokens, for
example with step_stopwords(), step_stem(),
step_tokenfilter(), or step_ngram(), and then
turn them into numeric features with step_tf(),
step_tfidf(), or step_texthash(). See
vignette("cookbook---using-more-complex-recipes-involving-text")
for complete examples.