-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check both MD5 locations for S3 KMS support. #1272
Open
neolynx
wants to merge
4
commits into
master
Choose a base branch
from
fix/kms-md5-check
base: master
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neolynx
force-pushed
the
fix/kms-md5-check
branch
from
April 17, 2024 21:22
194d629
to
52cbd46
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1272 +/- ##
==========================================
+ Coverage 74.79% 74.81% +0.02%
==========================================
Files 144 144
Lines 16256 16261 +5
==========================================
+ Hits 12158 12165 +7
Misses 3156 3156
+ Partials 942 940 -2 ☔ View full report in Codecov by Sentry. |
neolynx
changed the title
fix/kms md5 check
Check both MD5 locations for S3 KMS support.
Apr 20, 2024
6 tasks
neolynx
force-pushed
the
fix/kms-md5-check
branch
from
April 21, 2024 09:27
75b68a3
to
bd8b595
Compare
neolynx
force-pushed
the
fix/kms-md5-check
branch
from
April 24, 2024 14:56
bd8b595
to
6a7d80c
Compare
If the S3 bucket used to house a repo has KMS encryption enabled then the etag of an object may not match the MD5 of the file. This may cause an incorrect error to be reported stating the file already exists and is different. A mechanism exists to work around this issue by using the MD5 stored in object metadata. This check doesn't always cover the case where KMS is enabled as the fallback is only used if the etag is not 32 characters long. This commit changes the fallback mechanism so that it is used in any case where the object's etag does not match the source MD5. This will incur a performance penalty of an extra head request for each object with a mismatch.
Adds check to see if the S3 bucket is encrypted by default. If so this uses the existing workaround for object etags not matching file MD5s.
neolynx
force-pushed
the
fix/kms-md5-check
branch
from
April 24, 2024 15:41
6a7d80c
to
4960b40
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces #1167
Fixes #1117
Description of the Change
If the S3 bucket used to house a repo has KMS encryption enabled then the etag of an object may not match the MD5 of the file. This may cause an incorrect error to be reported stating the file already exists and is different.
A mechanism exists to work around this issue by using the MD5 stored in object metadata. This check doesn't always cover the case where KMS is enabled as the fallback is only used if the etag is not 32 characters long.
This commit changes the fallback mechanism so that it is used in any case where the object's etag is not 32 characters or if the S3 bucket has encryption enabled for new objects by default.
Checklist
AUTHORS