Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

devtools::check() running examples with --run-donttest when run_dont_test = FALSE #2216

Closed
HaydenMacDonald opened this issue Apr 26, 2020 · 6 comments
Closed

devtools::check() running examples with --run-donttest when run_dont_test = FALSE #2216

HaydenMacDonald opened this issue Apr 26, 2020 · 6 comments
Labels
feature a feature request or enhancement

Comments

@HaydenMacDonald
Copy link

HaydenMacDonald commented Apr 26, 2020 •
edited by jimhester

devtools::check() seems to be ignoring run_dont_test = FALSE resulting in this output:

√  checking examples (4.4s)
/   checking examples with --run-donttest ...

I'm unsure as to whether this has anything to do with the R 4.0.0 update, but rcmdcheck doesn't seem to have any open related issues.

Session info 
Session info ------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 4.0.0 (2020-04-24)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_Canada.1252         
 ctype    English_Canada.1252         
 tz       America/New_York            
 date     2020-04-26                  

Packages ---------------------------------------------------------------------------------------------------
 ! package       * version date       lib source        
   acepack         1.4.1   2016-10-29 [1] CRAN (R 4.0.0)
   assertthat      0.2.1   2019-03-21 [1] CRAN (R 4.0.0)
   backports       1.1.6   2020-04-05 [1] CRAN (R 4.0.0)
   base64enc       0.1-3   2015-07-28 [1] CRAN (R 4.0.0)
   callr           3.4.3   2020-03-28 [1] CRAN (R 4.0.0)
   checkmate       2.0.0   2020-02-06 [1] CRAN (R 4.0.0)
   cli             2.0.2   2020-02-28 [1] CRAN (R 4.0.0)
   clipr           0.7.0   2019-07-23 [1] CRAN (R 4.0.0)
   cluster         2.1.0   2019-06-19 [2] CRAN (R 4.0.0)
   colorspace      1.4-1   2019-03-18 [1] CRAN (R 4.0.0)
   crayon          1.3.4   2017-09-16 [1] CRAN (R 4.0.0)
   data.table      1.12.8  2019-12-09 [1] CRAN (R 4.0.0)
   desc            1.2.0   2018-05-01 [1] CRAN (R 4.0.0)
   devtools      * 2.3.0   2020-04-10 [1] CRAN (R 4.0.0)
   digest          0.6.25  2020-02-23 [1] CRAN (R 4.0.0)
   dplyr           0.8.5   2020-03-07 [1] CRAN (R 4.0.0)
   ellipsis        0.3.0   2019-09-20 [1] CRAN (R 4.0.0)
   evaluate        0.14    2019-05-28 [1] CRAN (R 4.0.0)
   fansi           0.4.1   2020-01-08 [1] CRAN (R 4.0.0)
   foreign         0.8-78  2020-04-13 [2] CRAN (R 4.0.0)
   Formula         1.2-3   2018-05-03 [1] CRAN (R 4.0.0)
   fs              1.4.1   2020-04-04 [1] CRAN (R 4.0.0)
   generics        0.0.2   2018-11-29 [1] CRAN (R 4.0.0)
   ggplot2         3.3.0   2020-03-05 [1] CRAN (R 4.0.0)
   glue            1.4.0   2020-04-03 [1] CRAN (R 4.0.0)
   gridExtra       2.3     2017-09-09 [1] CRAN (R 4.0.0)
   gtable          0.3.0   2019-03-25 [1] CRAN (R 4.0.0)
   here            0.1     2017-05-28 [1] CRAN (R 4.0.0)
   Hmisc           4.4-0   2020-03-23 [1] CRAN (R 4.0.0)
   htmlTable       1.13.3  2019-12-04 [1] CRAN (R 4.0.0)
   htmltools       0.4.0   2019-10-04 [1] CRAN (R 4.0.0)
   htmlwidgets     1.5.1   2019-10-08 [1] CRAN (R 4.0.0)
   httr            1.4.1   2019-08-05 [1] CRAN (R 4.0.0)
   janitor         2.0.1   2020-04-12 [1] CRAN (R 4.0.0)
   jpeg            0.1-8.1 2019-10-24 [1] CRAN (R 4.0.0)
   knitr           1.28    2020-02-06 [1] CRAN (R 4.0.0)
   lattice         0.20-41 2020-04-02 [2] CRAN (R 4.0.0)
   latticeExtra    0.6-29  2019-12-19 [1] CRAN (R 4.0.0)
   lifecycle       0.2.0   2020-03-06 [1] CRAN (R 4.0.0)
   lubridate       1.7.8   2020-04-06 [1] CRAN (R 4.0.0)
   magrittr        1.5     2014-11-22 [1] CRAN (R 4.0.0)
   Matrix          1.2-18  2019-11-27 [2] CRAN (R 4.0.0)
   memoise         1.1.0   2017-04-21 [1] CRAN (R 4.0.0)
   munsell         0.5.0   2018-06-12 [1] CRAN (R 4.0.0)
   naniar          0.5.0   2020-02-28 [1] CRAN (R 4.0.0)
   nnet            7.3-13  2020-02-25 [2] CRAN (R 4.0.0)
   pillar          1.4.3   2019-12-20 [1] CRAN (R 4.0.0)
   pkgbuild        1.0.7   2020-04-25 [1] CRAN (R 4.0.0)
   pkgconfig       2.0.3   2019-09-22 [1] CRAN (R 4.0.0)
   pkgload         1.0.2   2018-10-29 [1] CRAN (R 4.0.0)
   plyr            1.8.6   2020-03-03 [1] CRAN (R 4.0.0)
   png             0.1-7   2013-12-03 [1] CRAN (R 4.0.0)
   polite          0.1.1   2019-11-30 [1] CRAN (R 4.0.0)
   prettyunits     1.1.1   2020-01-24 [1] CRAN (R 4.0.0)
   processx        3.4.2   2020-02-09 [1] CRAN (R 4.0.0)
   ps              1.3.2   2020-02-13 [1] CRAN (R 4.0.0)
   purrr           0.3.4   2020-04-17 [1] CRAN (R 4.0.0)
   R6              2.4.1   2019-11-12 [1] CRAN (R 4.0.0)
   ratelimitr      0.4.1   2018-10-07 [1] CRAN (R 4.0.0)
   rcmdcheck       1.3.3   2019-05-07 [1] CRAN (R 4.0.0)
   RColorBrewer    1.1-2   2014-12-07 [1] CRAN (R 4.0.0)
   Rcpp            1.0.4.6 2020-04-09 [1] CRAN (R 4.0.0)
   remotes         2.1.1   2020-02-15 [1] CRAN (R 4.0.0)
   reprex        * 0.3.0   2019-05-16 [1] CRAN (R 4.0.0)
   rlang           0.4.5   2020-03-01 [1] CRAN (R 4.0.0)
   rmarkdown       2.1     2020-01-20 [1] CRAN (R 4.0.0)
   robotstxt       0.6.2   2018-07-18 [1] CRAN (R 4.0.0)
   roxygen2        7.1.0   2020-03-11 [1] CRAN (R 4.0.0)
   rpart           4.1-15  2019-04-12 [2] CRAN (R 4.0.0)
   rprojroot       1.3-2   2018-01-03 [1] CRAN (R 4.0.0)
   rstudioapi      0.11    2020-02-07 [1] CRAN (R 4.0.0)
   rvest           0.3.5   2019-11-08 [1] CRAN (R 4.0.0)
   scales          1.1.0   2019-11-18 [1] CRAN (R 4.0.0)
   sessioninfo     1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
   snakecase       0.11.0  2019-05-25 [1] CRAN (R 4.0.0)
 P squashinformr * 0.1.0   2020-04-26 [?] local         
   stringi         1.4.6   2020-02-17 [1] CRAN (R 4.0.0)
   stringr         1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
   survival        3.1-12  2020-04-10 [2] CRAN (R 4.0.0)
   testthat        2.3.2   2020-03-02 [1] CRAN (R 4.0.0)
   tibble          3.0.1   2020-04-20 [1] CRAN (R 4.0.0)
   tidyr           1.0.2   2020-01-24 [1] CRAN (R 4.0.0)
   tidyselect      1.0.0   2020-01-27 [1] CRAN (R 4.0.0)
   usethis       * 1.6.0   2020-04-09 [1] CRAN (R 4.0.0)
   vctrs           0.2.4   2020-03-10 [1] CRAN (R 4.0.0)
   visdat          0.5.3   2019-02-15 [1] CRAN (R 4.0.0)
   whisker         0.4     2019-08-28 [1] CRAN (R 4.0.0)
   withr           2.2.0   2020-04-20 [1] CRAN (R 4.0.0)
   xfun            0.13    2020-04-13 [1] CRAN (R 4.0.0)
   xml2            1.3.2   2020-04-23 [1] CRAN (R 4.0.0)
   xopen           1.0.0   2018-09-17 [1] CRAN (R 4.0.0)

[1] C:/Users/Hayden/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.0/library

 P -- Loaded and on-disk path mismatch.
</details>
@HaydenMacDonald
Copy link
Author

HaydenMacDonald commented Apr 27, 2020 •
edited

I've just noticed that R 4.0.0 has changed R CMD check --as-cran to run \donttest examples (see Utilities section here). "This can be temporarily circumvented during development by setting environment variable _R_CHECK_DONTTEST_EXAMPLES_ to a false value."

@jimhester jimhester added the feature a feature request or enhancement label May 1, 2020
@maelle maelle mentioned this issue May 12, 2020
Closed
jefferis added a commit to natverse/rcatmaid that referenced this issue May 14, 2020
@jefferis
Don't run donttest examples on travis
f9e7603 
* see e.g. r-lib/devtools#2216
* if this package were on a CRAN we'd have a bigger problem
dblodgett-usgs added a commit to dblodgett-usgs/nhdplusTools that referenced this issue May 17, 2020
@dblodgett-usgs
https://github.com/r-lib/devtools/issues/2216
73f1a0a
jefferis added a commit to natverse/elmr that referenced this issue Jun 8, 2020
@jefferis
Don't run donttest examples on travis
869179c 
* R 4.0 now (stupidly IMHO) runs these by default (see r-lib/devtools#2216)
@jefferis jefferis mentioned this issue Jun 8, 2020
Merged
SridharJagannathan added a commit to natverse/rcatmaid that referenced this issue Jun 8, 2020
@SridharJagannathan
Updated temporary fix (due to r version 4 issue) to not run tests see (…
7872dcb 
…r-lib/devtools#2216)
@Alternikaner
Copy link

Alternikaner commented Jun 9, 2020 •
edited

In case it isn't obvious to anyone landing here how/where to set the environmental variable mentioned above (it wasn't clear to me) you can try the following:

  1. Create a text file named check.Renviron containing the following line:
    R_CHECK_DONTTEST_EXAMPLES=false
  2. Put this file in the (~/.R/) directory [Can be identified with normalizePath("~/.R/") if required]

@hadley hadley closed this as completed in a795ae4 Jul 30, 2020
@hadley
Copy link
Member

hadley commented Jul 30, 2020

I clarified the docs. Generally, now if you don't want to run tests on CRAN \dontrun{} is more likely to work, but using \dontrun{} may cause initial submission to fail.

@marouenbg marouenbg mentioned this issue Oct 15, 2020
Merged
@maelle maelle mentioned this issue Nov 13, 2020
Merged
kelly-sovacool added a commit to SchlossLab/mikropml that referenced this issue Nov 20, 2020
@kelly-sovacool
Don't run \donttest{} examples on gh-actions
681741c 
Behavior of devtools::check() changed in R 4.0
r-lib/devtools#2216

CRAN maintainers prefer \donttest{} for long-running examples over \dontrun{}
https://stackoverflow.com/questions/12038160/how-to-not-run-an-example-using-roxygen2
@dankelley dankelley mentioned this issue Mar 24, 2021
Closed
@dankelley
Copy link

@hadley, I wonder whether you might say a bit more here about your statement "but using \dontrun{} may cause initial submission to fail" in the just-previous comment. Why would using \dontrun{} cause a problem?

This is a general question, but in case it's of any interest, I am asking because we have some examples in the oce package (dankelley/oce#1798) that are display how to work with data that are not provided in the package. For years, we have used \donttest{} to wrap such examples.

I think the choices as

  1. use \donttest{} ... this seems likely to cause CRAN rejection because the CRAN test machine will not be able to run the code.
  2. use \dontrun{} ... this seems to be the way forward, but your note about submission failing worries me (hence this request)
  3. move example code to \preformatted{} blocks, probably within the Details section of the documentation
  4. drop the example code

As a user, I often jump to the Examples part of documentation, so I think option 3 is undesirable, although not as bad as option 4. Since option 1 seems destined to fail in the present-day R system, we are left with option 2.

Option 2 is what I infer as the best solution, based on https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Marking-text and the purpose of this comment is to ask whether this is correct (in light of your proviso about rejection).

My guess is that quite a few folks come upon this issue through web searches, and so it might be very helpful to get a bit of advice posted here. Even a single phrase like "Use \dontrun{} for examples using data not provided with your package" would be enormously helpful to those of us who fear CRAN rejection.

@hadley
Copy link
Member

hadley commented Mar 24, 2021

@dankelley on initial submission (only) CRAN tends to complain if you use \dontrun{}.

@dankelley
Copy link

@hadley thanks very much for the clarification. It helps a lot. Dan.

@KevCaz KevCaz mentioned this issue Oct 30, 2021
Closed
@SridharJagannathan SridharJagannathan mentioned this issue Jul 5, 2022
Closed
@phuongquan phuongquan mentioned this issue Oct 16, 2022
Closed
29 tasks
@brookslogan brookslogan mentioned this issue Oct 25, 2022
Merged
31 tasks
@palmjulia palmjulia mentioned this issue Nov 14, 2022
Closed
@howardbaek howardbaek mentioned this issue Jan 19, 2023
Closed
fhui28 added a commit to fhui28/CBFM that referenced this issue Feb 27, 2023
@fhui28
Switching all dontest to dontrun; see r-lib/devtools#2216
0e5356b
@dankelley dankelley mentioned this issue Apr 26, 2023
Closed
@oganm oganm mentioned this issue Jul 25, 2023
Closed
mauricio-tki added a commit to malaria-atlas-project/malariaAtlas that referenced this issue Oct 5, 2023
@mauricio-tki
Prevent running flaky examples on CRAN
dbb9eb5 
See also r-lib/devtools#2216
@diraol diraol mentioned this issue Oct 15, 2023
Closed
zekrom-vale added a commit to zekrom-vale/ShinySQLBrowser that referenced this issue Apr 6, 2024
@zekrom-vale
Well, dontrun is ignored
8ff8443 
r-lib/devtools#2216
zekrom-vale added a commit to zekrom-vale/ShinySQLBrowser that referenced this issue Apr 6, 2024
@zekrom-vale
Well, dontrun is ignored
c4009ff 
r-lib/devtools#2216
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature a feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hadley @dankelley @jimhester @Alternikaner @HaydenMacDonald