Add go module and cache support to project ops #894
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've chosen to use standard go tooling environment variables from the host environment. This provides a more intuitive/automatical workflow for users (like me) who have customization in their workflow to support switching between different versions of go. Without GOPATH or GOCACHE set, this will default to a `.go` directory at the root of the project, which means it won't conflict with anything global for the average developer. This removes the dependency on the HOME variable, which I like, since that can access a pretty broad set of sensitive things. It also avoids the issues brought up in Remitly/mothra#168 This speeds up local development with dev ops signficantly. A rebuild of the cli with no local code changes is 87% faster. <details> Here are subsequent builds of the cli (I'm using an opctl build from my `mini-opctl` branch, which is why the output looks different and has the timing info). ``` cameronlittle@Chinook:~/Developer/opctl (±go-module-caching) (⬢ v12.18.1) (go1.15.8) (⎈preprod.k8s.remitly.com) ❯ GOPATH="" GOCACHE="" opctl run ./cli/.opspec/compile [27937cf3 ./cli/.opspec/compile] started op [febdff65 ./cli/.opspec/compile] started docker.io/library/golang:1.15 [796c389a ./cli/.opspec/compile] started docker.io/library/golang:1.15 [febdff65 ./cli/.opspec/compile] 1.15: Pulling from library/golang [796c389a ./cli/.opspec/compile] 1.15: Pulling from library/golang [febdff65 ./cli/.opspec/compile] Digest: sha256:194804c9c296dd70c95e0a7f7e97e82e6b37078ea9bff0637e39dceec90464fe [febdff65 ./cli/.opspec/compile] Status: Image is up to date for golang:1.15 [796c389a ./cli/.opspec/compile] Digest: sha256:194804c9c296dd70c95e0a7f7e97e82e6b37078ea9bff0637e39dceec90464fe [796c389a ./cli/.opspec/compile] Status: Image is up to date for golang:1.15 [796c389a ./cli/.opspec/compile] go: downloading github.com/gorilla/handlers v1.4.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/gorilla/mux v1.7.3 [796c389a ./cli/.opspec/compile] go: downloading github.com/blang/semver v3.5.1+incompatible [febdff65 ./cli/.opspec/compile] go: downloading github.com/rakyll/statik v0.1.7-0.20191104211043-6b2f3ee522b6 [febdff65 ./cli/.opspec/compile] go: downloading github.com/pkg/errors v0.9.1 [febdff65 ./cli/.opspec/compile] go: downloading k8s.io/api v0.19.1 [febdff65 ./cli/.opspec/compile] go: downloading k8s.io/apimachinery v0.19.1 [febdff65 ./cli/.opspec/compile] go: downloading k8s.io/client-go v0.19.1 [796c389a ./cli/.opspec/compile] go: downloading k8s.io/client-go v0.19.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/pkg/errors v0.9.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 [796c389a ./cli/.opspec/compile] go: downloading github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 [796c389a ./cli/.opspec/compile] go: downloading github.com/rhysd/go-github-selfupdate v1.2.3 [febdff65 ./cli/.opspec/compile] go: downloading github.com/blang/semver v3.5.1+incompatible [796c389a ./cli/.opspec/compile] go: downloading github.com/docker/docker v17.12.0-ce-rc1.0.20200916142827-bd33bbf0497b+incompatible [febdff65 ./cli/.opspec/compile] go: downloading github.com/peterh/liner v1.1.0 [796c389a ./cli/.opspec/compile] go: downloading golang.org/x/net v0.0.0-20200707034311-ab3426394381 [796c389a ./cli/.opspec/compile] go: downloading k8s.io/api v0.19.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/gorilla/mux v1.7.3 [febdff65 ./cli/.opspec/compile] go: downloading github.com/docker/docker v17.12.0-ce-rc1.0.20200916142827-bd33bbf0497b+incompatible [febdff65 ./cli/.opspec/compile] go: downloading github.com/rhysd/go-github-selfupdate v1.2.3 [796c389a ./cli/.opspec/compile] go: downloading k8s.io/apimachinery v0.19.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/peterh/liner v1.1.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/fatih/color v1.7.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/go-git/go-git/v5 v5.2.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/jawher/mow.cli v1.1.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/docker/go-connections v0.4.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/containers/image/v5 v5.0.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/google/go-github/v30 v30.1.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/xeipuuv/gojsonschema v1.2.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/gorilla/websocket v1.4.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/opencontainers/go-digest v1.0.0-rc1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/google/go-querystring v1.0.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/dgraph-io/badger/v2 v2.0.3 [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-utils/lockfile v0.0.0-20170803195317-342df9650a96 [febdff65 ./cli/.opspec/compile] go: downloading github.com/vbauerster/mpb v3.4.0+incompatible [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-utils/pscanary v0.0.0-20170803195345-167b86ee2e7e [febdff65 ./cli/.opspec/compile] go: downloading github.com/tcnksm/go-gitconfig v0.1.2 [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-interfaces/ios v0.0.0-20170803194714-da59acb78efc [febdff65 ./cli/.opspec/compile] go: downloading golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad [febdff65 ./cli/.opspec/compile] go: downloading github.com/mattn/go-isatty v0.0.10 [febdff65 ./cli/.opspec/compile] go: downloading github.com/inconshreveable/go-update v0.0.0-20160112193335-8152e7eb6ccf [febdff65 ./cli/.opspec/compile] go: downloading github.com/mattn/go-colorable v0.1.2 [796c389a ./cli/.opspec/compile] go: downloading github.com/opencontainers/image-spec v1.0.2-0.20190823105129-775207bd45b6 [febdff65 ./cli/.opspec/compile] go: downloading k8s.io/klog/v2 v2.2.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/VividCortex/ewma v1.1.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/go-logr/logr v0.2.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-utils/filecopier v0.0.0-20170803193939-16f96e9dcff4 [796c389a ./cli/.opspec/compile] go: downloading golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 [febdff65 ./cli/.opspec/compile] go: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 [febdff65 ./cli/.opspec/compile] go: downloading github.com/appdataspec/sdk-golang v0.0.0-20170917062448-0c0ade7a92f7 [febdff65 ./cli/.opspec/compile] go: downloading github.com/ulikunitz/xz v0.5.9 [febdff65 ./cli/.opspec/compile] go: downloading github.com/etcd-io/bbolt v1.3.3 [796c389a ./cli/.opspec/compile] go: downloading github.com/google/gofuzz v1.1.0 [796c389a ./cli/.opspec/compile] go: downloading golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 [febdff65 ./cli/.opspec/compile] go: downloading github.com/jfbus/httprs v0.0.0-20180614100821-7861a1150846 [febdff65 ./cli/.opspec/compile] go: downloading github.com/klauspost/compress v1.8.6 [febdff65 ./cli/.opspec/compile] go: downloading github.com/satori/go.uuid v0.0.0-20181028125025-b2ce2384e17b [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-interfaces/ihttp v0.0.0-20170731143308-228dd9eedf13 [796c389a ./cli/.opspec/compile] go: downloading github.com/sergi/go-diff v1.1.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/sethgrid/pester v0.0.0-20180430140037-03e26c9abbbf [796c389a ./cli/.opspec/compile] go: downloading github.com/go-git/go-billy/v5 v5.0.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/dustin/go-humanize v1.0.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/mitchellh/copystructure v1.0.0 [febdff65 ./cli/.opspec/compile] go: downloading golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e [796c389a ./cli/.opspec/compile] go: downloading github.com/dgraph-io/ristretto v0.0.2 [febdff65 ./cli/.opspec/compile] go: downloading golang.org/x/term v0.0.0-20201117132131-f5c789dd3221 [796c389a ./cli/.opspec/compile] go: downloading github.com/golang/protobuf v1.4.2 [febdff65 ./cli/.opspec/compile] go: downloading github.com/docker/distribution v2.7.1+incompatible [796c389a ./cli/.opspec/compile] go: downloading gopkg.in/inf.v0 v0.9.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-utils/dircopier v0.0.0-20170803194507-75bc9e581ed2 [796c389a ./cli/.opspec/compile] go: downloading github.com/containers/libtrust v0.0.0-20190913040956-14b96171aa3b [796c389a ./cli/.opspec/compile] go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.0.1 [febdff65 ./cli/.opspec/compile] go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.0.1 [febdff65 ./cli/.opspec/compile] go: downloading sigs.k8s.io/yaml v1.2.0 [796c389a ./cli/.opspec/compile] go: downloading google.golang.org/protobuf v1.24.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/golang/snappy v0.0.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/json-iterator/go v1.1.10 [febdff65 ./cli/.opspec/compile] go: downloading github.com/mitchellh/go-homedir v1.1.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/davecgh/go-spew v1.1.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/ghodss/yaml v1.0.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/gogo/protobuf v1.3.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/imdario/mergo v0.3.9 [febdff65 ./cli/.opspec/compile] go: downloading github.com/modern-go/reflect2 v1.0.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/containers/storage v1.13.5 [796c389a ./cli/.opspec/compile] go: downloading github.com/sirupsen/logrus v1.4.2 [febdff65 ./cli/.opspec/compile] go: downloading github.com/golang-interfaces/iioutil v0.0.0-20170803194630-7d1c0886acdc [febdff65 ./cli/.opspec/compile] go: downloading github.com/mitchellh/reflectwalk v1.0.1 [febdff65 ./cli/.opspec/compile] go: downloading github.com/klauspost/pgzip v1.2.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/gogo/protobuf v1.3.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 [796c389a ./cli/.opspec/compile] go: downloading github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 [796c389a ./cli/.opspec/compile] go: downloading github.com/go-git/gcfg v1.5.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/cespare/xxhash v1.1.0 [febdff65 ./cli/.opspec/compile] go: downloading github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c [796c389a ./cli/.opspec/compile] go: downloading github.com/mattn/go-runewidth v0.0.3 [796c389a ./cli/.opspec/compile] go: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190809123943-df4f5c81cb3b [796c389a ./cli/.opspec/compile] go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd [febdff65 ./cli/.opspec/compile] go: downloading golang.org/x/text v0.3.5 [796c389a ./cli/.opspec/compile] go: downloading github.com/modern-go/reflect2 v1.0.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/emirpasic/gods v1.12.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/googleapis/gnostic v0.4.1 [796c389a ./cli/.opspec/compile] go: downloading google.golang.org/grpc v1.27.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c [796c389a ./cli/.opspec/compile] go: downloading gopkg.in/yaml.v2 v2.2.8 [796c389a ./cli/.opspec/compile] go: downloading github.com/docker/go-units v0.4.0 [796c389a ./cli/.opspec/compile] go: downloading github.com/golang-interfaces/github.com-gorilla-websocket v0.0.0-20190604222234-f7c71d63fecb [796c389a ./cli/.opspec/compile] go: downloading golang.org/x/time v0.0.0-20191024005414-555d28b269f0 [796c389a ./cli/.opspec/compile] go: downloading github.com/containers/storage v1.13.5 [febdff65 ./cli/.opspec/compile] go: downloading github.com/xanzy/ssh-agent v0.2.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/golang-interfaces/iioutil v0.0.0-20170803194630-7d1c0886acdc [796c389a ./cli/.opspec/compile] go: downloading github.com/containerd/containerd v1.2.7 [796c389a ./cli/.opspec/compile] go: downloading gopkg.in/warnings.v0 v0.1.2 [796c389a ./cli/.opspec/compile] go: downloading github.com/klauspost/pgzip v1.2.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/mitchellh/reflectwalk v1.0.1 [796c389a ./cli/.opspec/compile] go: downloading golang.org/x/text v0.3.5 [febdff65 ./cli/.opspec/compile] go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd [796c389a ./cli/.opspec/compile] go: downloading github.com/klauspost/cpuid v1.2.1 [febdff65 ./cli/.opspec/compile] go: downloading k8s.io/utils v0.0.0-20201104234853-8146046b121e [796c389a ./cli/.opspec/compile] go: downloading k8s.io/utils v0.0.0-20201104234853-8146046b121e [796c389a ./cli/.opspec/compile] go: downloading google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 [796c389a ./cli/.opspec/compile] go: downloading github.com/xanzy/ssh-agent v0.2.1 [796c389a ./cli/.opspec/compile] go: downloading github.com/opencontainers/runc v1.0.0-rc8 [febdff65 ./cli/.opspec/compile] go: downloading github.com/opencontainers/runc v1.0.0-rc8 [796c389a ./cli/.opspec/compile] go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd [febdff65 ./cli/.opspec/compile] go: downloading google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 [febdff65 ./cli/.opspec/compile] docker.io/library/golang:1.15 exited [796c389a ./cli/.opspec/compile] docker.io/library/golang:1.15 exited [27937cf3 ./cli/.opspec/compile] op succeeded ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ◎ ☑ ./cli/.opspec/compile 2m25.816331s ├─◉ ☑ parallel loop 2m25.808346s ├─◉ ☑ febdff65 docker.io/library/golang:1.15 2m25.177632s go build -o cli/opctl-linux-amd64 ./cli └─◉ ☑ 796c389a docker.io/library/golang:1.15 2m25.808009s go build -o cli/opctl-darwin-amd64 ./cli cameronlittle@Chinook:~/Developer/opctl (±go-module-caching) (⬢ v12.18.1) (go1.15.8) (⎈preprod.k8s.remitly.com) ❯ GOPATH="" GOCACHE="" opctl run ./cli/.opspec/compile [f2af80cb ./cli/.opspec/compile] started op [9776f242 ./cli/.opspec/compile] started docker.io/library/golang:1.15 [284f0c1f ./cli/.opspec/compile] started docker.io/library/golang:1.15 [9776f242 ./cli/.opspec/compile] 1.15: Pulling from library/golang [284f0c1f ./cli/.opspec/compile] 1.15: Pulling from library/golang [9776f242 ./cli/.opspec/compile] Digest: sha256:194804c9c296dd70c95e0a7f7e97e82e6b37078ea9bff0637e39dceec90464fe [9776f242 ./cli/.opspec/compile] Status: Image is up to date for golang:1.15 [284f0c1f ./cli/.opspec/compile] Digest: sha256:194804c9c296dd70c95e0a7f7e97e82e6b37078ea9bff0637e39dceec90464fe [284f0c1f ./cli/.opspec/compile] Status: Image is up to date for golang:1.15 [9776f242 ./cli/.opspec/compile] docker.io/library/golang:1.15 exited [284f0c1f ./cli/.opspec/compile] docker.io/library/golang:1.15 exited [f2af80cb ./cli/.opspec/compile] op succeeded ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ ◎ ☑ ./cli/.opspec/compile 19.09475s ├─◉ ☑ parallel loop 19.085537s ├─◉ ☑ 9776f242 docker.io/library/golang:1.15 18.637741s go build -o cli/opctl-linux-amd64 ./cli └─◉ ☑ 284f0c1f docker.io/library/golang:1.15 19.085083s go build -o cli/opctl-darwin-amd64 ./cli ``` </details> Resolves #787
|
@jaredru, you might be interested in this, as I want to apply it to mothra as well |
|
I might be seeing poor ide performance with the .go directory in the repo. I think it's picking it up for indexing. This is also sort of happening when checking gofmt in the test op. |
|
One possible feature for opctl that could fix this would be might be to support a first-class concept of a cache directory that opctl manages in it's |
Codecov Report
@@ Coverage Diff @@
## main #894 +/- ##
=======================================
Coverage 64.27% 64.27%
=======================================
Files 165 165
Lines 5744 5744
=======================================
Hits 3692 3692
Misses 1829 1829
Partials 223 223 Continue to review full report at Codecov.
|
|
As you know, I like this direction overall. My biggest concern is Linux. Because opctl doesn't do the "magic" Forge used to to run as the current user, your |
|
Thanks for the reminder about that. That's a good reason to not use the standard |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've chosen to use standard go tooling environment variables from the host environment. This provides a more intuitive/automatical workflow for users (like me) who have customization in their workflow to support switching between different versions of go. Without GOPATH or GOCACHE set, this will default to a
.godirectory at the root of the project, which means it won't conflict with anything global or modify random state for the average developer. This removes the dependency on the HOME variable, which I like, since that can access a pretty broad set of sensitive things. It also avoids the issues brought up in https://github.com/Remitly/mothra/pull/168This speeds up local development with dev ops signficantly. A rebuild of the cli with no local code changes is 87% faster.
Here are subsequent builds of the cli (I'm using an opctl build from my
mini-opctlbranch, which is why the output looks different and has the timing info).Resolves #787