Skip to contents

Development guidelines for quanteda

Source: AGENTS.md

Scope and approach

These instructions apply to the entire repository. Follow existing code and documentation conventions, and assess proposed changes critically before implementing them. Preserve public behaviour unless a change is intentional and documented.

Project overview

quanteda is an R package for the quantitative analysis of textual data. Its core objects and operations cover corpora and document variables, tokenisation and token manipulation, dictionaries, document-feature matrices (dfm), feature-co-occurrence matrices (fcm), and supporting text-analysis utilities.

The package combines R and Rcpp/C++ code. Changes to object construction, external-pointer tokens, indexing, or serialization can affect both layers and should be tested across the relevant object types.

Repository structure

  • R/: R source, S3 methods, validators, and user-facing APIs.
  • src/: Rcpp/C++ implementation.
  • inst/include/: installed C++ headers used by quanteda and extensions.
  • tests/testthat/: unit and regression tests.
  • tests/data/: test fixtures; tests/data_creation/ contains data-generation scripts.
  • man/: roxygen2-generated help files; do not edit these manually.
  • vignettes/: package vignettes and long-form documentation.
  • vignettes/pkgdown/: source material used for the package website.
  • _pkgdown.yml: pkgdown navigation and configuration.
  • docs/: generated pkgdown site.
  • data/: packaged example data.
  • DESCRIPTION: package metadata, version, dependencies, and collation order.
  • NAMESPACE: generated by roxygen2; do not edit manually.
  • .github/workflows/: R CMD check, coverage, and R-hub workflows.

R conventions

  • Use two-space indentation and no tabs.
  • Use snake_case for functions, arguments, and objects, following nearby code.
  • Keep high-level functions above their helpers where practical.
  • Prefer clear base R code and namespace-qualified calls for non-imported functions.
  • Match the repository’s existing error-handling patterns, including validators, message_error(), and stop(). Do not introduce a new error framework without an intentional project-wide decision.
  • Use British English in prose, consistent with Language: en-GB in DESCRIPTION.
  • End text files with a newline.

Documentation and generated files

  • Document exported functions with roxygen2, including complete @param and @return sections and useful examples where appropriate.
  • Add @export or the appropriate S3 registration tags for public APIs.
  • Run devtools::document() after changing roxygen comments.
  • Review generated changes to man/ and NAMESPACE; do not edit them directly.
  • After changing an Rcpp-exported interface, run Rcpp::compileAttributes() and review the generated changes to R/RcppExports.R and src/RcppExports.cpp.
  • Treat docs/ as generated output. Change _pkgdown.yml, roxygen source, or vignette source first, then regenerate the site with pkgdown::build_site().
  • Add new reference topics to _pkgdown.yml when they need explicit placement.
  • Add a concise NEWS.md bullet for user-visible changes. Put function names in code font and reference the issue or PR and contributor when applicable.

Testing

  • Add focused regression tests for bug fixes and tests for new behaviour.
  • Place tests with the most closely related existing tests; source and test files do not need a strict one-to-one naming relationship.
  • Cover typical inputs, edge cases, errors, empty objects, and relevant object representations.
  • Keep tests deterministic and avoid network access.
  • Start with the narrowest relevant test file, then run the broader suite in proportion to the change.

Useful commands from a clean R session include:

devtools::test(filter = "tokens")
devtools::test(reporter = "check")
covr::package_coverage()

Do not claim a particular testthat edition unless it is explicitly configured in DESCRIPTION.

Building and checking

Use clean, reproducible R sessions where practical:

R --quiet --vanilla

For normal development checks:

devtools::document()
devtools::test(reporter = "check")
devtools::check()

For a CRAN-style command-line check:

R CMD build .
R CMD check --as-cran quanteda_*.tar.gz

Review the tarball name if multiple local builds exist.

For website changes:

pkgdown::check_pkgdown()
pkgdown::build_site()

C++ and Rcpp changes

  • Keep the R and C++ representations of quanteda objects consistent.
  • Treat code in inst/include/ as a potentially public C++ interface.
  • Check ownership, copying, and mutation semantics carefully for external pointers.
  • Test both ordinary tokens and tokens_xptr paths when shared behaviour is affected.
  • Run package compilation and the relevant tests after modifying headers or files under src/.

Git and pull requests

  • Keep changes scoped and preserve unrelated work already present in the tree.
  • Use concise, imperative commit subjects, for example Fix token type lookup.
  • Base pull requests on master unless the task specifies another branch.
  • In a PR description, explain the motivation, user-visible behaviour, main implementation changes, tests, and documentation updates.
  • Confirm the GitHub Actions R CMD check and coverage results for changes that affect package code.

Security and reproducibility

  • Never commit API keys, access tokens, credentials, or private data.
  • Do not make unit tests depend on external services.
  • Record relevant R and package versions when diagnosing environment-specific failures.

Before finishing

  • Inspect the final diff for accidental generated or unrelated changes.
  • Run the checks appropriate to the risk and scope of the change.
  • Report what was tested and anything that could not be tested.
  • Do not describe a suggestion as an improvement until its trade-offs and fit with quanteda have been evaluated.