Returns a document by feature matrix reduced in size based on document and term frequency, usually in terms of a minimum frequency, but may also be in terms of maximum frequencies. Setting a combination of minimum and maximum frequencies will select features based on a range.

Feature selection is implemented by considering features across all documents, by summing them for term frequency, or counting the documents in which they occur for document frequency. Rank and quantile versions of these are also implemented, for taking the first $$n$$ features in terms of descending order of overall global counts or document frequencies, or as a quantile of all frequencies.

dfm_trim(x, min_termfreq = NULL, max_termfreq = NULL,
termfreq_type = c("count", "prop", "rank", "quantile"),
min_docfreq = NULL, max_docfreq = NULL, docfreq_type = c("count",
"prop", "rank", "quantile"), sparsity = NULL,
verbose = quanteda_options("verbose"), ...)

## Arguments

x a dfm object minimum/maximum values of feature frequencies across all documents, below/above which features will be removed how min_termfreq and max_termfreq are interpreted. "count" sums the frequencies; "prop" divides the term frequencies by the total sum; "rank" is matched against the inverted ranking of features in terms of overall frequency, so that 1, 2, ... are the highest and second highest frequency features, and so on; "quantile" sets the cutoffs according to the quantiles (see quantile) of term frequencies. minimum/maximum values of a feature's document frequency, below/above which features will be removed specify how min_docfreq and max_docfreq are interpreted. "count" is the same as docfreq(x, scheme = "count"); "prop" divides the document frequencies by the total sum; "rank" is matched against the inverted ranking of document frequency, so that 1, 2, ... are the features with the highest and second highest document frequencies, and so on; "quantile" sets the cutoffs according to the quantiles (see quantile) of document frequencies. equivalent to 1 - min_docfreq, included for comparison with tm print messages not used

## Value

A dfm reduced in features (with the same number of documents)

## Note

Trimming a dfm object is an operation based on the values in the document-feature matrix. To select subsets of a dfm based on the features themselves (meaning the feature labels from featnames) -- such as those matching a regular expression, or removing features matching a stopword list, use dfm_select.

## See also

dfm_select, dfm_sample

## Examples

(dfmat <- dfm(data_corpus_inaugural[1:5]))#> Document-feature matrix of: 5 documents, 1,948 features (69.5% sparse).
# keep only words occurring >= 10 times and in >= 2 documents
dfm_trim(dfmat, min_termfreq = 10, min_docfreq = 2)#> Document-feature matrix of: 5 documents, 107 features (16.3% sparse).
# keep only words occurring >= 10 times and in at least 0.4 of the documents
dfm_trim(dfmat, min_termfreq = 10, min_docfreq = 0.4)#> Document-feature matrix of: 5 documents, 107 features (16.3% sparse).
# keep only words occurring <= 10 times and in <=2 documents
dfm_trim(dfmat, max_termfreq = 10, max_docfreq = 2)#> Document-feature matrix of: 5 documents, 1,675 features (76.5% sparse).
# keep only words occurring <= 10 times and in at most 3/4 of the documents
dfm_trim(dfmat, max_termfreq = 10, max_docfreq = 0.75)#> Document-feature matrix of: 5 documents, 0 features.
#> 5 x 0 sparse Matrix of class "dfm"
#>                  features
#> docs
#>   1789-Washington
#>   1793-Washington
#>   1797-Adams
#>   1801-Jefferson
#>   1805-Jefferson
# keep only words occurring 5 times in 1000, and in 2 of 5 of documents
dfm_trim(dfmat, min_docfreq = 0.4, min_termfreq = 0.005, termfreq_type = "prop")#> Document-feature matrix of: 5 documents, 30 features (7.33% sparse).
# keep only words occurring frequently (top 20%) and in <=2 documents
dfm_trim(dfmat, min_termfreq = 0.2, max_docfreq = 2, termfreq_type = "quantile")#> Document-feature matrix of: 5 documents, 1,676 features (76.5% sparse).
# NOT RUN {
# compare to removeSparseTerms from the tm package
(dfmattm <- convert(dfmat, "tm"))
tm::removeSparseTerms(dfmattm, 0.7)
dfm_trim(dfmat, min_docfreq = 0.3)
dfm_trim(dfmat, sparsity = 0.7)
# }