Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IGNORE RDIFF & testing examples in CRAN release #538

Open
jarioksa opened this issue Oct 5, 2022 · 0 comments
Open

IGNORE RDIFF & testing examples in CRAN release #538

jarioksa opened this issue Oct 5, 2022 · 0 comments

Comments

@jarioksa
Copy link
Contributor

jarioksa commented Oct 5, 2022

We have had reference output for vegan examples for a decade. In principle, it is useful as it can spot unwanted changes and regression in the results of functions. However, it is of limited use because several results are numerically unstable, and comparison against reference output always gives many screenfuls of output hiding output from regressions. For this reason we have also removed the reference output from CRAN releases. For some time R has enabled marking numerically unstable segments so that these are not compared to the reference output, and this could allow having more useful detection of other changes. Such sections can be marked adding special comments to the examples in man pages:

## IGNORE_RDIFF_BEGIN
scores(cca(dune)) # signs can flip in eigenvector ordination
## IGNORE_RDIFF_END

Numerically unstable sections are mainly in two main groups:

  1. Last decimals or almost-zero values can differ in last bits depending on hardware, operating systems, R version, variant of BLAS etc.
  2. Sings of eigenvectors are arbitrary, and in addition to reasons above, it seems that these can change by phase of moon and positions of trams in Helsinki network (see sporalogy).

In proper tests these would be handled allowing fuzz in numerical comparison or using absolute values of eigenvectors, but the examples are primarily examples, although they can be used as additional tests as well.

I have now started marking these sections in man pages. The current reference output (in vegan/tests/Examples/vegan-Ex.Rout.save) is based on my tests with Apple Silicon M1 & M2 and Apple's native BLAS ("neural engine"). The first markings were based on running tests with the same hardware and software except using R reference BLAS, and testing vegan in WinBuilder strains of R (and it seems that ordisurf.Rd still needs a bit more extensive marking). I am not sure that the marking is yet sufficient to have the reference output in the next CRAN release, but we can start making reference output useful to all developers and mark the unstable sections. Naturally, it is good to understand why the section is unstable to avoid marking sections in examples when we should fix code. It can also make sense to to change example, but we must remember that they are primarily examples and should be user-friendly and readable.

Random number generator is a potential source of numerical differences. However, this is something that we should avoid marking out. In principle, same seed should give the same random number sequences in all versions of R (since 3.6-0) in all platforms, and if there are differences, the reasons should be analysed instead of marking out.

NB., even if we do not include reference output of examples, the examples are run in CRAN checks and in github, and these tests will detect errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant