New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add shellcheck for verifying shell scripts #1552
Add shellcheck for verifying shell scripts #1552
Conversation
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
Welcome @Vandit1604! |
Hi @Vandit1604. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
ce997c7
to
8ad17ce
Compare
/ok-to-test |
Hey @mimowo Should I install shellcheck via |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Vandit1604 I would suggest adding a Make target to install shellcheck via the released binary: https://github.com/koalaman/shellcheck/releases/tag/v0.9.0 so that we can perform the verification locally.
Which warnings generated by shellcheck will we ignore? |
Because the current stacktrace is filled with many warnings we can suppress the ones that can be disregarded. find . -type f \( -name '*.sh' \) ! -path '*/vendor/*' | xargs ./shellcheck-v0.9.0/shellcheck -x -s sh
|
The make target works fine locally. Does the testing suite not wait for the wget command to download the binary? |
Makefile
Outdated
| wget -qi - | ||
# extracting tar.gz file | ||
tar -xvf shellcheck-v0.9.0.linux.x86_64.tar.xz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we download the binary to $PROJECT_DIR/bin
as well as other tools?
Lines 297 to 301 in 4593b42
PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) | |
GOLANGCI_LINT = $(PROJECT_DIR)/bin/golangci-lint | |
.PHONY: golangci-lint | |
golangci-lint: ## Download golangci-lint locally if necessary. | |
@GOBIN=$(PROJECT_DIR)/bin GO111MODULE=on $(GO_CMD) install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2 |
And then, should we call the $PROJECT_DIR/bin/shellcheck
in the Make shell-lint
target?
Makefile
Outdated
# removing previous shellcheck files if shellcheck give errors and warnings it wouldn't remove the downloaded files | ||
rm -rf shellcheck-v0.9.0* | ||
@echo "Installing ShellCheck" | ||
curl -s https://api.github.com/repos/koalaman/shellcheck/releases/latest \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should have such an automation logic.
Should we just download a binary using https://github.com/koalaman/shellcheck/releases/download/v0.9.0/shellcheck-v0.9.0.darwin.x86_64.tar.xz?
Makefile
Outdated
rm -rf shellcheck-v0.9.0* | ||
@echo "Installing ShellCheck" | ||
curl -s https://api.github.com/repos/koalaman/shellcheck/releases/latest \ | ||
| grep "browser_download_url.*linux.x86_64.tar.xz" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you obtain the OS and CPU arch from go env, and then download the binary for dedicated the OS and CPU arch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can use https://github.com/dvershinin/lastversion to download the latest stable release. It downloads the binary release specific to the OS.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that we should use the latest version. We should use a specified version as well as other tools.
Here, the specified version means v0.9.0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The utility downloads the latest stable version not the stable version. I tried getting the URL according to the OS and architecture but it didn't work.
Oh, there are many warnings :( |
I believe so, though it would take about two days. Are we not ignoring any warnings or errors? |
Ah. Finally got it. The test is failing locally too but after giving the output of shellcheck. As the line which is giving the error is originating from the same line
|
Makefile
Outdated
shell-lint: | $(PROJECT_BIN) | ||
cd $(PROJECT_BIN); \ | ||
if [ ! -d "shellcheck" ]; then \ | ||
DOWNLOAD_URL=$$(curl -s https://api.github.com/repos/koalaman/shellcheck/releases/latest | grep "browser_download_url.*linux.x86_64.tar.xz" | cut -d : -f 2,3 | tr -d \" ); \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we statically set the download URL, not dynamically?
I meant like the following:
GOOS = $(shell go env GOOS)
GOARCH = $(shell go env GOARCH)
...
ifeq ($(GOOS),arm64)
OS = aarch64
else
OS = $(GOOS)
endif
shell-lint:
wget -q -O /bin/shellcheck.tar.gz https://github.com/koalaman/shellcheck/releases/download/v0.9.0/shellcheck-v0.9.0.$(OS).$(GOARCH).tar.xz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the late response. I tried creating the URL statically
This is the output of GOOS and GOARCH on my computer
>> go env GOOS
linux
>> go env GOARCH
amd64
This is the URL I got but it didn't download the binary as we expected.
https://github.com/koalaman/shellcheck/releases/download/v0.9.0/shellcheck-v0.9.0.linux.amd64.tar.xz
I'll try something like this
GOOS = $(shell go env GOOS)
GOARCH = $(shell go env GOARCH)
ifeq ($(GOOS),darwin)
URL := https://github.com/koalaman/shellcheck/releases/download/v0.9.0/shellcheck-v0.9.0.darwin.x86_64.tar.xz
else
ARCH = aarch64
ifeq ($(GOARCH),arm64)
else
ARCH = armv6hf
endif
# another condition
endif
.PHONY: shell-lint
shell-lint:
wget -q -O /bin/shellcheck.tar.gz $(URL)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have pushed some changes downloading the release binary statically.
Makefile
Outdated
@@ -186,12 +186,29 @@ run-test-multikueue-e2e-%: FORCE | |||
@echo Running multikueue e2e for k8s ${K8S_VERSION} | |||
E2E_KIND_VERSION="kindest/node:v$(K8S_VERSION)" KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) CREATE_KIND_CLUSTER=$(CREATE_KIND_CLUSTER) ARTIFACTS="$(ARTIFACTS)/$@" IMAGE_TAG=$(IMAGE_TAG) ./hack/multikueue-e2e-test.sh | |||
|
|||
PROJECT_BIN := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST))))/bin |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have already defined in
Line 314 in 1a482fd
PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried using PROJECT_BIN but the target was returning error for undefined variable. I'll try again.
ea3c636
to
c16ad57
Compare
Cleaned up the commit history where I tried downloading shellcheck statically. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly lgtm
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you remove shell-check
from Makefile?
verify: gomod-verify vet ci-lint fmt-verify toc-verify manifests generate update-helm generate-apiref prepare-release-branch
Apologies. Might've removed it accidentally while cleaning up the git history. |
No worries, finding your mistake is my responsibility :) |
Hmm. There's no actual error message logging, but the tests fail. |
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
After running Edit: nvm, ran |
Alright, let us wait for the result of CI. |
All passed! /lgtm |
LGTM label has been added. Git tree hash: c3ca2c9e54d37eafddb23436f5657ca443317b82
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tenzen-y, Vandit1604 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Thanks @tenzen-y for the kind words. I'll start working addressing disabled errors soon. |
There is #wg-batch. |
* create a seperate shellscript for shellcheck and enclose complete paths whereever i haven't and remove unused variables * allow docker cli * change year and revert change in run-test.sh * disable issues that will be taken care of in another PR * enclose some variables in quotes * Remove unnecessary env Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * Update hack/verify-shellcheck.sh Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> * convert to single line export and declaration * add shell-lint in verify rule * execute shellcheck recursivly from ROOT of repository * Update hack/verify-shellcheck.sh Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com> --------- Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
What type of PR is this?
/kind feature
maybe?
What this PR does / why we need it:
We will be able to verify shell scripts using
make verify
Which issue(s) this PR fixes:
Fixes #1518
Special notes for your reviewer:
Upon running "make verify locally," numerous warnings such as these were displayed. Will we cater to them or ignore them?
Does this PR introduce a user-facing change?