Disable multi-part upload chunked encoding?

[jeff-lynch]

Greetings. I'm new to Soto and have hit a roadblock trying to use it for multi-part upload of a pre-signed URL to an internally created, supposedly S3-compatible, API.

Using the standard S3 client's multipartUpload method, our server is telling me that "chunked uploads are not supported, disable chunked encoding".

A Java client of that service uses the following to configure the client.

S3Configuration s3Configuration = S3Configuration.builder().chunkedEncodingEnabled(false).build();
S3Client client = s3ClientBuilder.serviceConfiguration(s3Configuration).build();

I'm not seeing a way to disable chunked encoding, but maybe I'm looking in the wrong place. I see the .allowChunkedStreaming option in AWSShapePayloadOptions, but I don't know how to put that to use (assuming that it is the correct option).

As a quick test, I modified SotoSignerV4/signer.swift to NOT add the x-amz-content-sha256 header and our server seems to like that, but clearly modifying the library isn't going to work in our CI environment.

I started down the path of duplicating large swaths of the Soto library, but that quickly turned into fighting with a bunch of internal structure and class dependencies.

Here's my current code that sets up and initiates the multipart upload:

        let endpoint = "\(scheme)://\(host)"
        let client = AWSClient(credentialProvider: .static(accessKeyId: "ACCESS", secretAccessKey: "SECRET"), httpClientProvider: .createNew)

        s3Client = S3(client: client, region: .useast1, endpoint: endpoint)
        let token = uploadInfo.uploadUrl.lastPathComponent
        let multipartUploadRequest = S3.CreateMultipartUploadRequest(bucket: "data", key: token)

        let filename = uploadInfo.fileUrl.path
        let uploadResponse = s3Client?.multipartUpload(multipartUploadRequest,
                                                       filename: filename,
                                                       progress: { progressPercent in
                                                        self.handleProgressWithPercentComplete(progressPercent)
                                                        })

Any pointers would be greatly appreciated. 🙏

[adam-fowler]

Ok there isn't an option to disable S3 chunked uploading at this point in time. I wouldn't be too hard to implement though. I don't know though if this is what they mean by chunked uploads, as there a number of different types of chunked upload.

Given I don't know if this would resolve your issue it would be good if you could test it locally first. Here are instructions to disable S3 chunked uploads

1. Fork both Soto and SotoCore
2. point your Soto to use your version of SotoCore in the Package.swift.
3. Edit this line in SotoCore https://github.com/soto-project/soto-core/blob/1f5878cee72ec47b30c42932a13a0b6ec4f73e6f/Sources/SotoCore/Message/AWSRequest.swift#L57 so it is never true.

Run your code. If this works I can easily add an option to disable it. You could even do it, if you would like to contribute.

[jeff-lynch]

I didn't do any of the forking, but just locally modified the file as you suggested, and that appears to have done the trick! Our server is happy with what we're sending.

To be honest, I'm not yet feeling qualified to contribute. Where would the option live? Would it be something I set on the S3 client?

[adam-fowler]

@jeff-lynch soto-project/soto-core#466

With this change you would create your S3 object as follows

let s3 = S3(
    client: awsClient, 
    region: .euwest1,
    options: .s3DisableChunkedUploads
)

[adam-fowler]

v5.7.2 of SotoCore has been released with the option to disable s3 chunked uploads