Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

examples/rgw: teach boto about the BLAKE3 checksum algorithm #57097

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

cbodley
Copy link
Contributor

@cbodley cbodley commented Apr 25, 2024

allow boto clients like awscli to use BLAKE3 as a checksum algorithm

$ aws s3api put-object help
...
       --checksum-algorithm (string)
          Indicates  the  algorithm used to create the checksum for the object
          when you use the SDK. This header will not  provide  any  additional
          functionality  if  you don't use the SDK. When you send this header,
          there must  be  a  corresponding  x-amz-checksum-*algorithm*  ``  or
          ``x-amz-trailer  header sent. Otherwise, Amazon S3 fails the request
          with the HTTP status code 400 Bad Request .
...
          Possible values:
          o CRC32
          o CRC32C
          o SHA1
          o SHA256
          o BLAKE3
...
       --checksum-blake3 (string)
          This header can be used as a data integrity check to verify that the
          data received is the same data that was originally sent. This header
          specifies the base64-encoded, 256-bit BLAKE3 checksum of the object.
          For more information, see Checking object integrity in the Amazon S3
          User Guide . This checksum algorithm is a Ceph RGW extension.
...
OUTPUT
...
       ChecksumBLAKE3 -> (string)
          The base64-encoded, 256-bit BLAKE3 checksum of the object. This will
          only be present if it was uploaded with the object.  With  multipart
          uploads,  this  may  not be a checksum value of the object. For more
          information about how checksums are calculated  with  multipart  up-
          loads,  see  Checking object integrity in the Amazon S3 User Guide .
          This checksum algorithm is a Ceph RGW extension.
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
Copy link
Contributor Author

cbodley commented Apr 25, 2024

i made this by copying all of the CRC32C pieces from https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json and renaming to BLAKE3

@cbodley cbodley added the rgw label Apr 25, 2024
@mattbenjamin
Copy link
Contributor

excellent! I'll try this out shortly; thanks for working on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants