Add travis-ci style pull request builder

[alex]

It'd be amazing for helping review documentation if RTD has a travis-ci pull request builder. It could watch for PRs/pushes, build docs for them, and then link them in the build status.

/cc @reaperhulk

[reaperhulk]

We do this with a jenkins builder right now on pyca/cryptography but if we could get it straight through RTD that would be wonderful.

[agjohnson]

This is on our radar, but we need to make some design decisions around how to handle arbitrary HTML uploads without allowing completely arbitrary HTML uploads.

Closing this as a duplicate of some bug I know exists and I'll find later :)

[ericholscher]

This isn't an arbitrary HTML upload ticket, it's a request to auto-build branches from PR's of projects that we know about. It's definitely something we should be doing.

[agjohnson]

The ticket description here seemed to describe building docs on Travis. Reopening this as we are describing a Github/Bitbucket pull request integration.

There are some design decisions that would be required here. Namely, how to handle pull requests from branches on forks given our currnet data model. We could easily support branches pushed to the repo we know about by expanding the webhook to handle PR open/update.

[alex]

This ticket isn't about building docs on travis; it's about building docs in the style of travis. That is, on every push for a PR, build the docs and link them so people can browse.

[bukzor]

If we could set up webhooks to tell readthedocs to build docs for pull requests that touch documentation, that would be ideal.

Currently when a pull request comes in that touches documentation, we have two distinctly distasteful options:

1. Eyeball the changes and merge it if it looks right, then try to remember to double-check rtfd when it builds the merged master.
2. Tell the requestor to set up rtfd for their branch. Even users that are intimately familiar with sphinx/readthedocs would often balk at this request, since it would have to be re-done each time the pull request used a unique branch name.

[ericholscher]

I agree this makes a lot of sense. We need to implement some kind of "temporary" version that would map to the PR, and clone from the remote repository sending the PR. Then we'd need to delete the version when the branch got merged, but I agree this would be quite useful.

[bukzor]

ericholscher: I think those can all be done based on data from the github webhooks, but I've no idea where the implementing code would live.I suppose it would be a new endpoint for the rtfd site?

[agjohnson]

Blocked on #2465

[stsewd]

I don't know how the GitHub API works for this case, but I think this feature #872 could be used for this

[ssbarnea]

Any news on this? I do build sphinx docs with Travis on multiple projects but it seems that even if building docs with travis may succeed, it may not succeed on RTD, causing more trouble.

As I find much easier to build docs myself, fully integrated into CI pipeline, I am starting to wonder what are the real benefits of using RTD in this case, as it seems to be a PITA.

Did anyone managed to find a workaround that would prevent people from merging PRs that would break building RTD documentation?

[stsewd]

@ssbarnea this is what you get from rtd for free 😉 https://docs.readthedocs.io/en/latest/features.html

I'm not sure how this feature would be integrated (rtd would have a lot of temporal docs), you can have something like this for checking your builds

https://github.com/rtfd/readthedocs.org/blob/e73b266558af84745dd1cfc9e9dec7aa3e091dde/tox.ini#L21-L24

And maybe using rstcheck (something like this #3624)

[thorade]

I would like to use this in combination with protected branches and status checks, so that PRs can only be merged if they do NOT break the building process on RTD.
https://help.github.com/articles/about-protected-branches/
https://help.github.com/articles/about-required-status-checks/
https://developer.github.com/v3/repos/statuses/

[KengoTODA]

to whom it may concern,

I created a GitHub Probot app that enables RTD build and share its document URL in PR. It may cover a part of your requirements, please check.

[ericholscher]

Interestingly, travis does it using the git aliases also: https://travis-ci.org/rtfd/readthedocs.org/jobs/482572527#L418

[davidfischer]

This feature is accepted and we plan to do it. Our plan is to do this after all build artifacts are stored in blob storage (see #5549 for a start on that) because this change will significantly increase our storage requirements as RTD would be storing build output for every PR docs build indefinitely. This will become higher priority in the latter half of 2019.

[alex]

That's wonderful to hear! If you ever want a beta tester, don't hesitate to holler!

[stsewd]

We have this feature in beta https://blog.readthedocs.com/building-docs-for-pull-requests/

[ericholscher]

Going to call this one closed, as it's now in the codebase. We will slowly roll it out, but it's done 👍

[jakirkham]

What are the steps for setting this up for a repo currently?

[stsewd]

@jakirkham you mean locally or on the site? On the site you need to request a beta access https://blog.readthedocs.com/building-docs-for-pull-requests/

For a local installation you need to enable this flag for your projects EXTERNAL_VERSION_BUILD

https://docs.readthedocs.io/en/stable/guides/feature-flags.html

You can do this from the admin of Features

[jakirkham]

Thanks for the detailed answer. Was meaning the site. Didn't know if things had changed in the past few months.

[flying-sheep]

Since I find this issue faster than the docs: https://docs.readthedocs.io/en/stable/guides/autobuild-docs-for-pull-requests.html