Naive Bayes classifier for texts

Fit a multinomial or Bernoulli Naive Bayes model, given a dfm and some training labels.

textmodel_nb(
  x,
  y,
  smooth = 1,
  prior = c("uniform", "docfreq", "termfreq"),
  distribution = c("multinomial", "Bernoulli")
)

Arguments

x	the dfm on which the model will be fit. Does not need to contain only the training documents.
y	vector of training labels associated with each document identified in train. (These will be converted to factors if not already factors.)
smooth	smoothing parameter for feature counts by class
prior	prior distribution on texts; one of "uniform", "docfreq", or "termfreq". See Prior Distributions below.
distribution	count model for text features, can be multinomial or Bernoulli. To fit a "binary multinomial" model, first convert the dfm to a binary matrix using dfm_weight(x, scheme = "boolean").

Value

textmodel_nb() returns a list consisting of the following (where \(I\) is the total number of documents, \(J\) is the total number of features, and \(k\) is the total number of training classes):

call

original function call

PwGc

\(k \times J\); probability of the word given the class (empirical likelihood)

Pc

\(k\)-length named numeric vector of class prior probabilities

PcGw

\(k \times J\); posterior class probability given the word

Pw

\(J \times 1\); baseline probability of the word

x

the \(I \times J\) training dfm x

y

the \(I\)-length y training class vector

distribution

the distribution argument

prior

the prior argument

smooth

the value of the smoothing parameter

Prior distributions

Prior distributions refer to the prior probabilities assigned to the training classes, and the choice of prior distribution affects the calculation of the fitted probabilities. The default is uniform priors, which sets the unconditional probability of observing the one class to be the same as observing any other class.

"Document frequency" means that the class priors will be taken from the relative proportions of the class documents used in the training set. This approach is so common that it is assumed in many examples, such as the worked example from Manning, Raghavan, and Schütze (2008) below. It is not the default in quanteda, however, since there may be nothing informative in the relative numbers of documents used to train a classifier other than the relative availability of the documents. When training classes are balanced in their number of documents (usually advisable), however, then the empirically computed "docfreq" would be equivalent to "uniform" priors.

Setting prior to "termfreq" makes the priors equal to the proportions of total feature counts found in the grouped documents in each training class, so that the classes with the largest number of features are assigned the largest priors. If the total count of features in each training class was the same, then "uniform" and "termfreq" would be the same.

References

Manning, C.D., Raghavan, P., & Schütze, H. (2008). An Introduction to Information Retrieval. Cambridge: Cambridge University Press (Chapter 13). Available at https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf.

Jurafsky, D. & Martin, J.H. (2018). From Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Draft of September 23, 2018 (Chapter 6, Naive Bayes). Available at https://web.stanford.edu/~jurafsky/slp3/.

See also

predict.textmodel_nb

Examples

## Example from 13.1 of _An Introduction to Information Retrieval_
txt <- c(d1 = "Chinese Beijing Chinese",
         d2 = "Chinese Chinese Shanghai",
         d3 = "Chinese Macao",
         d4 = "Tokyo Japan Chinese",
         d5 = "Chinese Chinese Chinese Tokyo Japan")
trainingset <- dfm(txt, tolower = FALSE)
trainingclass <- factor(c("Y", "Y", "Y", "N", NA), ordered = TRUE)

## replicate IIR p261 prediction for test set (document 5)
(tmod1 <- textmodel_nb(trainingset, y = trainingclass, prior = "docfreq"))
#> 
#> Call:
#> textmodel_nb.dfm(x = trainingset, y = trainingclass, prior = "docfreq")
#> 
#> Distribution: multinomial; prior: docfreq; smoothing value: 1; 4 training documents; 6 fitted features.
summary(tmod1)
#> 
#> Call:
#> textmodel_nb.dfm(x = trainingset, y = trainingclass, prior = "docfreq")
#> 
#> Class Priors:
#> (showing first 2 elements)
#>    N    Y 
#> 0.25 0.75 
#> 
#> Estimated Feature Scores:
#>   Chinese Beijing Shanghai  Macao  Tokyo  Japan
#> N  0.1474  0.2059   0.2059 0.2059 0.5091 0.5091
#> Y  0.8526  0.7941   0.7941 0.7941 0.4909 0.4909
coef(tmod1)
#>           classes
#> features           N         Y
#>   Chinese  0.1473684 0.8526316
#>   Beijing  0.2058824 0.7941176
#>   Shanghai 0.2058824 0.7941176
#>   Macao    0.2058824 0.7941176
#>   Tokyo    0.5090909 0.4909091
#>   Japan    0.5090909 0.4909091
predict(tmod1)
#> d1 d2 d3 d4 d5 
#>  Y  Y  Y  N  Y 
#> Levels: N Y

# contrast with other priors
predict(textmodel_nb(trainingset, y = trainingclass, prior = "uniform"))
#> d1 d2 d3 d4 d5 
#>  Y  Y  Y  N  N 
#> Levels: N Y
predict(textmodel_nb(trainingset, y = trainingclass, prior = "termfreq"))
#> d1 d2 d3 d4 d5 
#>  Y  Y  Y  N  Y 
#> Levels: N Y

## replicate IIR p264 Bernoulli Naive Bayes
tmod2 <- textmodel_nb(trainingset, y = trainingclass, distribution = "Bernoulli",
                        prior = "docfreq")
predict(tmod2, newdata = trainingset[5, ])
#> d5 
#>  N 
#> Levels: N Y