[Feature]: Rest API Plugin: provide option to upload binary data (multimedia files)

[sumitsum]

Is there an existing issue for this?

• I have searched the existing issues

Summary

Currently REST API plugin does not seem to have any option that allows users to directly transmit binary data / file content .
e.g. Postman provides a binary tab:
https://theappsmith.slack.com/files/U035SFJHW3A/F06S5N2BUTX/image.png

Why should this be worked on?

This should take away a lot of confusion related with file uploads using REST API query.

Front conversations

[sumitsum]

APMU, binary data upload is completely non functional via REST API app. Reasons:

1. On the server side dynamic binding data is converted from binary to either a UTF-8 or ISO_8859_1 string, both of which could lead to binary data corruption. To maintain the sanctity of binary data, it should be either stored as byte array or converted to base64.
2. Needs to be checked : does the client try to convert any binary data to String format before sending it to the server - if so then binary data must be converted to base64 string before sending it server. Converting it to any other char set format String e.g. utf-8 could cause data corruption.

Relevant code sections:

protected Mono<Param> parseExecuteParameter(Part part, AtomicLong totalReadableByteCount) {
        final Param param = new Param();
        param.setPseudoBindingName(part.name());
        return DataBufferUtils.join(part.content()).map(dataBuffer -> {
            byte[] bytes = new byte[dataBuffer.readableByteCount()];
            totalReadableByteCount.addAndGet(dataBuffer.readableByteCount());
            dataBuffer.read(bytes);
            DataBufferUtils.release(dataBuffer);
            param.setValue(new String(bytes, StandardCharsets.UTF_8));
            return param;
        });
    }

protected Mono<Void> parseExecuteBlobs(
            Flux<Part> partsFlux, ExecuteActionDTO dto, AtomicLong totalReadableByteCount) {
        Map<String, String> blobMap = new HashMap<>();
        dto.setBlobValuesMap(blobMap);

        return partsFlux
                .flatMap(part -> {
                    return DataBufferUtils.join(part.content()).map(dataBuffer -> {
                        byte[] bytes = new byte[dataBuffer.readableByteCount()];
                        totalReadableByteCount.addAndGet(dataBuffer.readableByteCount());
                        dataBuffer.read(bytes);
                        DataBufferUtils.release(dataBuffer);
                        blobMap.put(part.name(), new String(bytes, StandardCharsets.ISO_8859_1));
                        return Mono.empty();
                    });
                })
                .then();
    }

A possible feature implementation to enable binary file upload:

1. Provide a new tab in the body section called binary. On selecting this tab the content type should auto set to applicaton/octet-stream
2. In the new binary body tab, any file data must be provided as base64 string.
3. On the server add special handling for application/octet-stream content type. It should assume that the body data is base64 string and decode it to byte array.

[sumitsum]

Reasons why this could be treated as high priority issue:

• This is a very basic and recurring use case that is expected via REST API .
• Postman has a separate section to upload binary files. Many Postman users expect to see something similar here as well.
• Until this issue is addressed we can expect to see this issue reported by users to A force from time to time - taking up valuable bandwidth from the Integrations team.
• At the moment Appsmith documentation mentions that binary files can be uploaded via REST API. This is not true at the moment. Hence, the documentation would have to be updated. However, explicitly mentioning that binary files upload is not support would look bad because this a basic use case.
  cc: @rohan-arthur @NilanshBansal

[NilanshBansal]

Mentioned this in the Sprint Planning Doc for us to plan it in the Next Sprint.
https://www.notion.so/appsmith/9-Apr-2024-43c9fc673b8d4b91b65826470ff1a761?pvs=4#f7234aeb1d5041eb978d795b47440751

[PrasannaPtools]

This a problem I was running into. I tried uploading a photoshop file through the file uploader to a dropbox endpoint and the resulting file gives me an error when I try to open it.

Context

• Self hosted on Digital Ocean Droplet
• Version Appsmith v1.13

Summary

I upload a small photoshop file (71 KB) onto the file uploader and click upload.

The triggered query, upon file upload, is a POST request to this endpoint ("https://content.dropboxapi.com/2/files/upload").

I set the data format to binary in the file uploader widget and in the query's body I refer to the uploading file as {{DropboxFilePicker.files[0].data}}.

After uploading I receive a larger file in dropbox (147 KB). When I try to open the uploaded file I get the following error.