New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: use hc-install
for TF downloads + constraints
#4494
base: main
Are you sure you want to change the base?
Conversation
…ethod for custom download URL's
hc-install
for default downloading - still currently old method for custom download URL'shc-install
for default URL TF downloads
hc-install
for default URL TF downloadshc-install
for default URL TF downloads
One downside of swapping out warrensbox for hc-install is that we lose the potential to reuse the same library to download opentofu versions. Is opentofu possible with hc-install or would we need to add additional code to support it? If additional code, would swapping to hc-install be worth it in the long run? |
@nitrocode I do not think installing OpenTofu with hc-install is possible unfortunately. There are a couple of things that come to my mind:
I believe hc-install is also open license, so from my understanding it's not out of the realm of possibility for OpenTofu to fork/adapt a verson for themselves. It also looks as though I don't think Atlantis should pigeon-hole itself into using the same methods for obtaining Terraform and it's forks given the divergences that could come in the future - if there's a tool designed for obtaining Terraform and it is the best way forward, why not use it, and use a more suited approach for OpenTofu?
There could be a discussion about using tenv, tfenv, terraform-switcher etc. to actually download Terraform? This is my first PR for Atlantis and I don't know the in's and out's too well yet so I'm more than happy to hear other opinions/views! |
no, we got Hashicorp approval to use terraform with Atlantis, we do not have that problem. |
Sorry if I wasn't clear - I don't mean the Hashicorp License change - I mean the inclusion of the license file in the packaged TF that made Atlantis have to constrain below TF 1.8.2. I'm not questioning Atlantis' perms to use TF, merely saying that the current system broke because of the inclusion of a License File. If anyones tooling is most likely to not break from changes to TF packaging, it's Hashicorps own installer that they use internally |
…n't have to do s many conversions
this is a bloody nightmare.... ( runt is over) I guess we will not be opposed to using the tool if it makes Atlantis maintenance easier, but we have been talking about removing this functionality entirely at some point so that we do not have to maintain this type of feature, which can be very brittle. Now, we are a package manager for Terraform and Opentofu installs, which adds complexity. |
@jamengual I would argue that this PR is one step closer to removing the package manager aspect from Atlantis - it shifts the constraints management and downloading over to e.g. Currently, Atlantis calls
|
Any reason why we can't tweak go-getter usage to deal with multiple files? I like
Any future zip file changes or additional files will be ignored. For the grab+extract, we could either let We would need to pay the maintenance cost of most (everything?) And if HC |
hc-install
for default URL TF downloadshc-install
for TF downloads + constraints
+1 on this approach. We'll need a difference in implementation for OpenTofu anyway. As far as I can tell there isn't a compatible endpoint for resolving versions like there is with https://releases.hashicorp.com/terraform. The install script provided with OpenTofu uses the GitHub releases, so should be relatively stable. I struggled to get go-getter to work with the archive with multiple files in it. It also looks like it wont support the verifying signing methods used by OpenTofu. As far as I can tell it only has support for checksums. |
See this page https://releases-page.opentofu-get.pages.dev/releases/ Ref
|
Awesome, thanks @nitrocode. I think my observation about the checksums may be more accurate. We don't seem to (I may have missed this too) get shasum files with the tofu releases. |
What
Replace
go-getter
usage withhc-install
usage for both default + custom URL TF downloads.Why
Currently, a URL based on system architecture etc is put together to produce a download link.
go-getter
'sGetFile
is then used to get the binary. This is broken with TF>= 1.8.2
as a License.txt file is also included. TF maintainer suggested looking intohc-install
instead.Changes
URL Downloads
This refactor uses
hc-install
to:terraform
by default so we rename the binary toterraform{version}
to match current convention)Custom URL's
For custom URL usage, the root URL must have the same structure as the default
releases.hashicorp.com
layout. e.g. it must have 2index.json
files; 1 in the project root and 1 in the version root. e.g.https://releases.hashicorp.com/terraform/index.json
https://releases.hashicorp.com/terraform/1.8.2/index.json
This is because hc-install parses these json files behind the scenes for it's logic.
Example setup can be seen in this commit: 39443b3
The URL's in the json can be kept as the default Hashicorp releases url's - if a custom
ApiBaseUrl
is set, hc-install will use that instead as the base URL - the URL in the JSON is ignored.Can be done via mirroring the website, or an approach similar to this to make one locally
Removing usage of other packages
Usage of
warrensbox/terraform-switcher
is removedValidVersionFormat
(ref)hc-install
Usage of
go-getter
'sGetFile
is also removed.Future support for OpenTofu
Whilst
hc-install
cannot be used to install OpenTofu, I personally don't see that as a good reason not to use it for Terraform.Due to possible divergences in the future (look at the License File in the zip change, unexpected), the same method of go-getter etc. may not be viable for both anyway - there may end up being a lot of conditionals for vendor-specific changes.
I think a better approach is to have different "retrievers" for the different softwares.
hc-install
for Terraform,go-getter
or some other approach for OpenTofu etc.Notes
hc-install
which adds support for Custom Download URL's. There is a PR I opened today for hc-install upstream (here) to add the support. Once that goes through, then thego.mod
can be changed back to the normal upstream version.Tests
Seems to pass Unit Tests and Integration Tests.
Local Testing
Ran Atlants + ngrok locally
Test: Remove <=1.8.2 constraint, test that hc-install correctly downloads it.
References