fix: REST API multimedia file upload issue fixed

[sneha122]

Description

If we use REST API action and file picker widget to upload any multimedia files (image, audio, video, pdf, xlsx), The file upload would be successful but file would get corrupted upon uploading. This was happening because file picker widget encodes this file to base64 format, and we were uploading this same base64 string using REST API url. Instead we should have decoded this base64 and then uploaded the file to retain the original contents of the file.

This PR fixes that issue by adding a new tab in body of the REST API action called binary, once we select this tab, we get autogenerated header for Content-Type: application/octet-stream, in this binary input field we can then provide base64 encoded file contents, the server then decodes the contents before triggering the respective REST API and uploading the file.

Steps to test the issue

1. Add a file picker widget on canvas
2. Select data format as Base64
3. Upload any of pdf, image, audio, video, xlsx in file picker
4. Create a REST API action using Dropbox upload API
5. Configure the API headers as mentioned in the documentation, also configure file name in the header correctly
6. Go to body tab, select binary and file contents in input box using binding like {{Filepicker1.files[0].data}}
7. Execute this API
8. Now go to your dropbox account and check the uploaded file, you should be able to successfully preview it

Fixes #32378
or
Fixes Issue URL

Warning

If no issue exists, please create an issue first, and check with the maintainers if the issue is valid.

Automation

/ok-to-test tags="@tag.Datasource"

🔍 Cypress test results

Tip

🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/8844375718
Commit: 3316290
Cypress dashboard url: Click here!

Communication

Should the DevRel and Marketing teams inform users about this change?

• Yes
• No

Summary by CodeRabbit

• New Features

  • Added support for binary file uploads in API requests, including handling of base64-encoded files.
  • Expanded content type options to include a new "BINARY" type for API requests.

• Tests

  • Implemented new tests to verify the functionality of binary file uploads with dynamic data binding.

• Bug Fixes

  • Ensured correct handling and auto-generation of headers for binary file types and form urlencoded data formats.

[coderabbitai]

Walkthrough

Walkthrough

This set of changes introduces support for binary data handling in various components of the application, including the API editor and server-side utilities. It involves adding new content types, enhancing UI components for file uploads, and improving data parsing capabilities to handle binary and base64-encoded files effectively.

Changes

Files	Changes
.../API_Auto_Generated_Header_spec.ts .../ApiPage.ts .../CommonApiConstants.ts .../APIEditor/PostBodyData.tsx	Added support for binary data handling, including new content types and UI components for file uploads.
.../DataUtils.java .../RestApiPluginTest.java	Enhanced server-side utilities and tests for handling and testing binary data uploads.

Possibly related issues

• [Feature]: Rest API Plugin: provide option to upload binary data (multimedia files) #32378: The changes in this PR introduce binary data upload functionality, which directly addresses the feature request for supporting binary data uploads in the REST API plugin.
• [Bug]: Unable to download octet stream using the built-in download function #30285: The handling of binary octet streams has been improved, which may resolve issues related to corrupted file downloads as described in this issue.
• [Bug]: Logic in GetDynamicStringSegments function does not split string segments correctly when processing binary data which leads to exceptions during REST API queries #29150: Enhancements in handling binary data might resolve issues with incorrect segmentation of JSON objects when processing binary data, as outlined in this issue.

---

Recent Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between b867f9c and 3316290.

Files selected for processing (6)

• app/client/cypress/e2e/Regression/ServerSide/ApiTests/API_Auto_Generated_Header_spec.ts (1 hunks)
• app/client/cypress/support/Pages/ApiPage.ts (1 hunks)
• app/client/src/constants/ApiEditorConstants/CommonApiConstants.ts (2 hunks)
• app/client/src/pages/Editor/APIEditor/PostBodyData.tsx (1 hunks)
• app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/restApiUtils/helpers/DataUtils.java (4 hunks)
• app/server/appsmith-plugins/restApiPlugin/src/test/java/com/external/plugins/RestApiPluginTest.java (1 hunks)

Additional comments not posted (8)

app/client/cypress/e2e/Regression/ServerSide/ApiTests/API_Auto_Generated_Header_spec.ts (2)

11-16: The test for verifying the auto-generated header for binary file type is correctly implemented.

---

18-18: The test for verifying the auto-generated header for form urlencoded data format is correctly implemented.

app/client/src/constants/ApiEditorConstants/CommonApiConstants.ts (2)

61-61: The addition of the BINARY enum value with the correct MIME type is properly implemented.

---

76-76: The addition of the BINARY MIME type to POST_BODY_FORMAT_OPTIONS is correctly implemented.

app/client/src/pages/Editor/APIEditor/PostBodyData.tsx (1)

144-161: The implementation of the new case for handling binary data uploads using a DynamicTextField is correctly done.

app/client/cypress/support/Pages/ApiPage.ts (1)

290-290: The addition of the BINARY content type to the list of sub-tabs is correctly implemented.

app/server/appsmith-interfaces/src/main/java/com/appsmith/external/helpers/restApiUtils/helpers/DataUtils.java (1)

162-177: The implementation of the parseMultimediaData method for handling multimedia data parsing from base64-encoded files is correctly done.

app/server/appsmith-plugins/restApiPlugin/src/test/java/com/external/plugins/RestApiPluginTest.java (1)

2600-2654: Ensure proper handling of base64 decoding in the test for binary file uploads.

The test method testBinaryFileUploadAPIWithMustacheBinding correctly sets up a scenario to test binary file uploads with base64 encoded data. However, ensure that the base64 decoding process is explicitly handled or mocked within the test setup to accurately simulate the environment and validate the functionality.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[sneha122]

/build-deploy-preview skip-tests=true

[github-actions]

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/8830816864.
Workflow: On demand build Docker image and deploy preview.
skip-tests: true.
env: ``.
PR: 32921.
recreate: .

[github-actions]

Deploy-Preview-URL: https://ce-32921.dp.appsmith.com

[NilanshBansal]

LGTM

[NilanshBansal]

@sneha122 this change needs a documentation update as Binary tab needs to be selected to upload files.
https://docs.appsmith.com/build-apps/how-to-guides/Send-Filepicker-Data-with-API-Requests

Can you please inform the documentation team about the same.

[sneha122]

@sneha122 this change needs a documentation update as Binary tab needs to be selected to upload files. https://docs.appsmith.com/build-apps/how-to-guides/Send-Filepicker-Data-with-API-Requests

Can you please inform the documentation team about the same.

@NilanshBansal Yes I will be informing them about this change

[PrasannaPtools]

Thanks!

I just need to update to the latest then?

[sneha122]

@PrasannaPtools Hi are you using self hosted version of appsmith? If yes you will have to wait for next update, post that you can update your version. I can inform you once that upgrade happens.

[PrasannaPtools]

Hi Sneha, We are self hosted, and that sounds good. I just updated mine to version v1.22.1. Let me know when the next update is available! Best Regards, Prasanna Thuraisingam

…

On Tue, 30 Apr 2024 at 03:45, sneha122 ***@***.***> wrote: @PrasannaPtools <https://github.com/PrasannaPtools> Hi are you using self hosted version of appsmith? If yes you will have to wait for next update, post that you can update your version. I can inform you once that upgrade happens. — Reply to this email directly, view it on GitHub <#32921 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHYGOCZMIYH2F34YD2BRAZ3Y75DZ7AVCNFSM6AAAAABGW3RX5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBUGYYTGMZSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>