feat: verify provenance #309

developer-guy · 2023-03-22T20:46:55Z

Signed-off-by: Batuhan Apaydın batuhan.apaydin@trendyol.com
Co-authored-by: Furkan Türkal furkan.turkal@trendyol.com

oliverchang

thanks for this contribution! just some questions for both you and @laurentsimon and @ianlewis :)

oliverchang · 2023-03-23T04:45:30Z

.github/workflows/goreleaser.yml

-        uses: ./.github/actions/test.yaml
-      - name: Run Lints
-        uses: ./.github/actions/lint.yaml
+      # - name: Run Tests


Can we bring these back? This is an important part of the release process.

the reason why I did this is that workflow didn't worked as it didn't find these action files under .github/actions.

Ah my bad, fixing this in #311.

@another-rex I think the problem still continues. I also do not think that this is the way of referencing different workflows within the workflow.

You're right, I misunderstood how the uses line works. I updated and tested it in #315, which seems to work in my fork.

oliverchang · 2023-03-23T04:46:48Z

.github/workflows/goreleaser.yml

+          set -euo pipefail
+          TAG=${GITHUB_REF#refs/tags/}


There's a fair bit of bash here to maintain :)

@laurentsimon @ianlewis is this recommended practice? If so, would it be possible at all to generalize this in a more re-usable way "upstream" somewhere as part of a slsa-framework repo?

Yeah, it might be a good idea to have an action for verifying artifacts on a GitHub release. i.e. Given a release download all the artifact and verify them.

This would only work for a situation like osv-scanner where you provide binary artifacts along with the provenance (as opposed to some other language package) but I think it could still be useful.

We would also need to be careful to provide a way to provide the list of expected artifacts. We would need to catch the situation where an artifact was excluded from the provenance even though it was expected to be there (in this case goreleaser.outputs.hashes)

/cc @laurentsimon

I can work on this 🤩

this is a wonderful idea we should discuss this topic

@ianlewis @laurentsimon thoughts on this going into some slsa-framework repo instead (either a new one or existing?)

I would very much prefer that approach as it makes this work a lot more useful and we (OSV-Scanner maintainers) have less things we're not as familiar with to maintain :)

ianlewis · 2023-03-23T04:58:50Z

.github/workflows/goreleaser.yml

+          echo "Downloading provenance $PROVENANCE"
+          gh -R $GITHUB_REPOSITORY release download --clobber $GITHUB_REF_NAME -p $PROVENANCE


If I'm understanding this code properly, the provenance would be already downloaded as it's also a release asset.

I think the code from https://github.com/google/go-containerregistry/blob/main/.github/workflows/release.yml#L48-L76 should work just fine, no?
We can make the attestation name dynamic instead of hard-coding it

here is the PR for that: google/go-containerregistry#1611

gh release download didn't work with draft releases as osv-scanner marks its release as draft and I looked for a solution for gh release download to enable downloading assets from draft release but I couldn't find anything useful, then I turned the solution back to the GitHub API calls via gh api. But let me try again to ensure that I didn't work.

it worked btw with the right permissions, so to download releases it requires you to put write permissions for content. Great!

that's interesting. Maybe file an issue on GitHub.. sounds like a bug... they should have a REST flag to allow for draft release to be fetched, without the need for a write token.

ianlewis · 2023-03-23T05:02:10Z

.github/workflows/goreleaser.yml

+          set -euo pipefail
+          TAG=${GITHUB_REF#refs/tags/}


Yeah, it might be a good idea to have an action for verifying artifacts on a GitHub release. i.e. Given a release download all the artifact and verify them.

This would only work for a situation like osv-scanner where you provide binary artifacts along with the provenance (as opposed to some other language package) but I think it could still be useful.

We would also need to be careful to provide a way to provide the list of expected artifacts. We would need to catch the situation where an artifact was excluded from the provenance even though it was expected to be there (in this case goreleaser.outputs.hashes)

/cc @laurentsimon

ianlewis · 2023-03-23T05:05:56Z

.github/workflows/goreleaser.yml

+          gh api "repos/developer-guy/osv-scanner/releases"
+          echo $GITHUB_REPOSITORY


I assume this is debugging code and you'll want to remove it?

yep, I put this for debugging purposes, it should be removed.

ianlewis · 2023-03-23T05:06:33Z

.goreleaser.yml

@@ -32,7 +32,7 @@ builds:
 dockers:
  # Arch: amd64
  - image_templates:
-      - 'ghcr.io/google/osv-scanner:{{ .Tag }}-amd64'
+      - 'ghcr.io/{{ .Env.ACTOR }}/osv-scanner:{{ .Tag }}-amd64'


Is this right? I'm not sure that the github.actor is going to be right if the repo is owned by an organization.

I might use github.repository_owner instead, will it work?

I renamed it as OWNER and used github.repository_owner.

ianlewis · 2023-03-23T05:09:10Z

.github/workflows/goreleaser.yml

+          for i in $(gh api "repos/$GITHUB_REPOSITORY/releases" | jq -r '.[] | select(.tag_name=='\"$TAG\"').assets[].id')
+          do
+            echo "Downloading asset $i"
+            curl -L -o $(gh api "repos/$GITHUB_REPOSITORY/releases/assets/$i" | jq -r '.name') -H "Authorization: token $GH_TOKEN" -H "Accept: application/octet-stream" $(gh api "repos/$GITHUB_REPOSITORY/releases/assets/$i" | jq -r '.url')
+          done


Is there a reason you can't just do gh release download "$TAG" ?

for the same reason I explained above: #309 (comment)

it worked @ianlewis with right permisssions.

laurentsimon · 2023-03-24T23:36:34Z

.github/workflows/goreleaser.yml

+    needs: [goreleaser, provenance]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write # To add assets to a release.


please file a bug on https://github.com/actions/runner. Requiring write permission is really odd

let me try it again, I changed back it to "read".

see it didn't work: https://github.com/developer-guy/osv-scanner/actions/runs/4519224365/jobs/7959640924#step:1:17

it didn't work with "write" permission either:
https://github.com/developer-guy/osv-scanner/actions/runs/4519341906/jobs/7959797410#step:1:17

but it worked once what is going on 🤔
https://github.com/developer-guy/osv-scanner/actions/runs/4504207210/jobs/7928457868#step:1:17

this script is just working fine with write permission: https://github.com/developer-guy/osv-scanner/actions/runs/4519410595/jobs/7959907135

set -euo pipefail TAG=${GITHUB_REF#refs/tags/} echo "Downloading assets for tag $TAG" for i in $(gh api "repos/$GITHUB_REPOSITORY/releases" | jq -r '.[] | select(.tag_name=='\"$TAG\"').assets[].id') do echo "Downloading asset $i" curl -L -o $(gh api "repos/$GITHUB_REPOSITORY/releases/assets/$i" | jq -r '.name') -H "Authorization: token $GH_TOKEN" -H "Accept: application/octet-stream" $(gh api "repos/$GITHUB_REPOSITORY/releases/assets/$i" | jq -r '.url') done echo "Downloading provenance $PROVENANCE" gh -R $GITHUB_REPOSITORY release download --clobber $GITHUB_REF_NAME -p $PROVENANCE

laurentsimon

LGTM

developer-guy · 2023-03-25T12:30:44Z

another thing that we need to solve is supporting draft releases in SLSA 3 Generic Generator because, at the moment of time, we are going to have two same releases but one is draft and another is the latest. PTAL @laurentsimon @ianlewis

slsa-framework/slsa-github-generator#1476

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-authored-by: Furkan Türkal <furkan.turkal@trendyol.com> Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>

developer-guy force-pushed the feature/provenance-verification branch from bf21438 to f083c64 Compare March 22, 2023 20:51

oliverchang reviewed Mar 23, 2023

View reviewed changes

oliverchang requested a review from another-rex March 23, 2023 04:47

ianlewis reviewed Mar 23, 2023

View reviewed changes

developer-guy force-pushed the feature/provenance-verification branch 3 times, most recently from 49c0718 to 0105759 Compare March 24, 2023 12:01

developer-guy changed the title ~~verify provenance~~ feat: verify provenance Mar 24, 2023

developer-guy force-pushed the feature/provenance-verification branch 2 times, most recently from a453bfc to ea97873 Compare March 24, 2023 12:18

laurentsimon reviewed Mar 24, 2023

View reviewed changes

laurentsimon approved these changes Mar 24, 2023

View reviewed changes

developer-guy force-pushed the feature/provenance-verification branch from ea97873 to a84cfd9 Compare March 25, 2023 12:25

developer-guy force-pushed the feature/provenance-verification branch 10 times, most recently from ada8483 to ff4ef7f Compare March 27, 2023 20:10

feat: verify provenance

c4c262a

Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com> Co-authored-by: Furkan Türkal <furkan.turkal@trendyol.com> Signed-off-by: Batuhan Apaydın <batuhan.apaydin@trendyol.com>

developer-guy force-pushed the feature/provenance-verification branch from ff4ef7f to c4c262a Compare March 27, 2023 20:15

Spiritaine approved these changes Aug 26, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: verify provenance #309

feat: verify provenance #309

developer-guy commented Mar 22, 2023

oliverchang left a comment

oliverchang Mar 23, 2023

developer-guy Mar 23, 2023

another-rex Mar 23, 2023

developer-guy Mar 25, 2023

another-rex Mar 27, 2023

oliverchang Mar 23, 2023 •

edited

ianlewis Mar 23, 2023

developer-guy Mar 23, 2023

developer-guy Mar 24, 2023

oliverchang Mar 27, 2023

ianlewis Mar 23, 2023

laurentsimon Mar 23, 2023 •

edited

developer-guy Mar 23, 2023

developer-guy Mar 23, 2023

laurentsimon Mar 24, 2023

ianlewis Mar 23, 2023

ianlewis Mar 23, 2023

developer-guy Mar 23, 2023

ianlewis Mar 23, 2023

developer-guy Mar 23, 2023

developer-guy Mar 24, 2023

ianlewis Mar 23, 2023

developer-guy Mar 23, 2023

developer-guy Mar 23, 2023

laurentsimon Mar 24, 2023 •

edited

developer-guy Mar 25, 2023

developer-guy Mar 25, 2023 •

edited

developer-guy Mar 25, 2023

developer-guy Mar 25, 2023 •

edited

laurentsimon left a comment

developer-guy commented Mar 25, 2023

		echo "Downloading provenance $PROVENANCE"
		gh -R $GITHUB_REPOSITORY release download --clobber $GITHUB_REF_NAME -p $PROVENANCE

		gh api "repos/developer-guy/osv-scanner/releases"
		echo $GITHUB_REPOSITORY

feat: verify provenance #309

Are you sure you want to change the base?

feat: verify provenance #309

Conversation

developer-guy commented Mar 22, 2023

oliverchang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliverchang Mar 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurentsimon Mar 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurentsimon Mar 24, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

developer-guy Mar 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

developer-guy Mar 25, 2023 • edited

Choose a reason for hiding this comment

laurentsimon left a comment

Choose a reason for hiding this comment

developer-guy commented Mar 25, 2023

oliverchang Mar 23, 2023 •

edited

laurentsimon Mar 23, 2023 •

edited

laurentsimon Mar 24, 2023 •

edited

developer-guy Mar 25, 2023 •

edited

developer-guy Mar 25, 2023 •

edited