This function reassign tokens IDs after changes in removal or transformation of tokens. It update token IDs to ensure that they are unique and dense. Usually, this function is only applied to tokens_xptr objects at the end of text pre-processing.
Arguments
- x
the tokens_xptr object.
- force
if
TRUE, update tokens IDs even when it may not be necessary.
Examples
toks <- tokens(c(one = "a b c d A B C D",
two = "A B C d"), xptr = TRUE)
toks <- tokens_tolower(toks)
types(toks)
#> [1] "a" "b" "c" "d"
types(tokens_recompile(toks))
#> [1] "a" "b" "c" "d"