Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client Side Encryption with S3EncryptionMultipartUploaderV2 fails on large files. #2512

Open
elt4n0 opened this issue Aug 29, 2022 · 1 comment
Assignees
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog

Comments

@elt4n0
Copy link

elt4n0 commented Aug 29, 2022

Describe the bug

When using S3EncryptionMultipartUploaderV2 with a file larger than 4GB a warning is thrown during encryption, and an exception is thrown during upload. This does not happen with smaller files.

Expected Behavior

Success on S3EncryptionMultipartUploaderV2::upload()

Current Behavior

When using S3EncryptionMultipartUploaderV2 with a file larger than 4GB this message appears:

PHP Warning: openssl_encrypt(): data is too long in libs\aws\aws-sdk-php\src\Crypto\AesGcmEncryptingStream.php on line 108

After a while, exception is thrown:

An exception occurred while completing a multipart upload: Error executing "CompleteMultipartUpload" on "encryption-test/lfmb_TESNICO.mp4?uploadId=345dd8b5-b470-4c92-8525-86298794e9e4"; AWS HTTP error: Client error: `POST encryption-test/lfmb_TESNICO.mp4?uploadId=345dd8b5-b470-4c92-8525-86298794e9e4` resulted in a `400 Bad Request` response:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed (truncated...)
 MalformedXML (client): The XML you provided was not well-formed or did not validate against our published schema. - <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema.</Message><Key>lfmb_TESNICO.mp4</Key><BucketName>encryption-test</BucketName><Resource>/encryption-test/lfmb_TESNICO.mp4</Resource><RequestId>170FDB5FFCA88848</RequestId><HostId>4bf9e65b-bbb0-488d-9da9-d54f60abb5c0</HostId></Error>

Reproduction Steps

$uploader = new S3EncryptionMultipartUploaderV2( 
         new Aws\S3\S3Client([
            'version' => 'latest',
            'region'  => 'us-east-1',
            'use_path_style_endpoint' => true,
            'credentials' => [
                    'key'    => $options['credentials']['key'],
                    'secret' => $options['credentials']['secret'],
                ]
        ]), ,fopen('path to a file larger than 4GB', 'r'),
            [
                '@MaterialsProvider' => $materialsProvider,
                '@CipherOptions' => [ 'Cipher' => 'gcm', 'KeySize' => 128 ],
                '@KmsEncryptionContext' => [],
                'bucket' => $this->bucket,
                'key' => 'thenamefile',
                'concurrency' => 5,
                'part_size' => null,
            ]
        );

$uploader->upload();

Possible Solution

openssl_encrypt is know have trouble with large files.
Maybe include a mechanism to encode by chunks.

Additional Information/Context

No response

SDK version used

"aws/aws-sdk-php": "^3.147"

Environment details (Version of PHP (php -v)? OS name and version, etc.)

PHP 7.4.30 (cli) (built: Jun 7 2022 15:36:03) ( NTS Visual C++ 2017 x64 )

@elt4n0 elt4n0 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 29, 2022
@yenfryherrerafeliz yenfryherrerafeliz self-assigned this Aug 29, 2022
@yenfryherrerafeliz yenfryherrerafeliz added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Aug 29, 2022
@yenfryherrerafeliz
Copy link
Contributor

Hi @elt4n0, thanks for opening this issue. I have reproduced the reported behavior and indeed the issue is coming from openssl library. I am going to leave this issue under review so we can address this further, and will provide updates accordingly.

Thanks!

@yenfryherrerafeliz yenfryherrerafeliz removed the needs-triage This issue or PR still needs to be triaged. label Sep 2, 2022
@yenfryherrerafeliz yenfryherrerafeliz removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Dec 7, 2022
@yenfryherrerafeliz yenfryherrerafeliz added the p2 This is a standard priority issue label Jan 2, 2023
@yenfryherrerafeliz yenfryherrerafeliz added queued This issues is on the AWS team's backlog and removed needs-review labels Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog
Projects
None yet
Development

No branches or pull requests

2 participants