Releases

Checklist

Available Spark versions

• Make sure the list of known Spark versions in inst/extdata/versions.json is up-to-date (NOTE: this is not really release-specific and should really be done every once in a while. Also, because by default, the main branch of sparklyr is used as the source-of-truth, this can be updated after a sparklyr release as well.)

Branch

• Create a feature/sparklyr-x.y branch and PR.

Sparklyr contributors: if you see 'git@github.com: Permission denied (publickey).' or similar while trying to push your local branch to upstream, try the following workaround instead:

git remote remove upstream
git remote add upstream https://github.com/sparklyr/sparklyr.git   # avoid git://.../sparklyr.git, use https instead for 'upstream'
mv ~/.ssh{,.old}   # make git not attempt to use any of your deployment keys while pushing -- there might be a better way of accomplishing this though but this would do
git push -u upstream 'feature/sparklyr-x.y.z'  # will need to log in using your github username and password instead when pushing
mv ~/.ssh{.old,}   # and then restore your ~/.ssh directory, obviously

Build

• Update versions in the DESCRIPTION file and NEWS file.
• Make sure all contributions are documented in NEWS file.
• Update default Livy Jars in livy_config(), see livy_connection.R#L568
• Re-generate docs/reference/*.html in the sparklyr repo using pkgdown (see more details about it here), check in the changes, and then follow instructions in https://github.com/rstudio/spark.rstudio.com to generate updated content about the release on spark.rstudio.com, and create a PR for this updated content.

Test

• Ensure Travis and AppVeyor tests are passing.
• Regenerate README.Rmd to validate main use cases using rmarkdown.
• Test yarn-client and yarn-cluster connection in EMR or similar service. Also, see https://github.com/sparklyr/sparklyr/wiki/EMR
• Test Livy connection in HDInsights or similar service. See https://github.com/sparklyr/sparklyr/wiki/HDInsights
• Test Databricks gateway connection in Databricks. Also, see https://github.com/sparklyr/sparklyr/wiki/Databricks

LF-AI announcement

• Prepare release announcement with summary/highlights of the release, link to release notes, and target release date and share with jserafin@linuxfoundation.org.

Release

Build official binary, test in CRAN and submit.

• Build package locally running R CMD build sparklyr
• Check packages is ready for CRAN with R CMD check --as-cran <sparklyr-package>
• Validate build with win-builder.r-project.org.
• Submit to CRAN with cran.r-project.org/submit.html.

Note: Last step requires email confirmation from current maintainer. If maintainer needs to change at some point and the current maintainer is not available, we would need to ask by email to CRAN to update the maintainer. Ideally, if there is a maintainer change, the maintainer name should be change in a CRAN submission.

Last Step

Assuming CRAN submission is successful, then tag the commit of the release with "CRAN v<major>.<minor>.<patch>" and update https://github.com/sparklyr/sparklyr/releases with the tag.