S3 Upload Plugin + thoughts on upload

[jeanbmar]

Hey there,

I've developed a simple plugin to upload files to S3: https://github.com/jeanbmar/payload-s3-upload

Update 08/17/2021

• add support for resized uploads
• ensure file safenames using database checks (as a replacement for local fs checks)

08/06/2021

• initial release

A few thoughts on upload:

• Saving files to disk should be optional. It shouldn't be the S3 or any other cloud plugin role to delete local files. I will try to hack around setting staticDir to /dev/null or something. (implemented in feat: allow completely disabling local file storage #272)
• Getting the ability to save files locally using an official plugin bundled with Payload would be great.
• Same applies to transformations. For instance, a resizing plugin would take the data buffer, generates an array of buffers with the different sizes, and pass it to the hook chain. As of now, the entire transformation logic would have to be recoded in S3 plugin. (implemented in feat: exposes auto-sized uploads on payload req #270)

Overall, the plugin system is easy to understand and works well. Good job!

[richardvanbergen]

Hello @jeanbmar

I thought you might be interested, I'm actually working on something like this myself, though it's less S3-specific.

https://github.com/richardvanbergen/payload-plugin-cloud-storage

Saving files to disk should be optional. It shouldn't be the S3 or any other cloud plugin role to delete local files. I will try to hack around setting staticDir to /dev/null or something.

The @jmikrut mentioned this is possible via hooks in this discussion, although I can't see where the local storage is disabled in the linked thread now I'm looking at it.

For instance, a resizing plugin would take the data buffer, generates an array of buffers with the different sizes, and pass it to the hook chain. As of now, the entire transformation logic would have to be recoded in S3 plugin.

Yes, there are a couple of gaps in the resize/upload pipeline to my mind. I've just yesterday started making the first steps to supporting sizes which as you say involves resizing the images a second time. I've bought this up in a discussion somewhere on here before but I can't find it at the moment.

There's also the ability to set null for one of the dimensions and the fit parameter for resized images.

If we can get those two things sorted then I think we're all good. For now, I'll be resizing the images myself.

[richardvanbergen]

I saw your plugin and there's promising stuff, but I had a hard time understanding the cloudStorageFields mechanics. It would be easier to manage those properties on the upload collections directly

Absolutely and you're free to do that! You can disable those fields and add it yourself.

The rationale behind the added fields is that it can add it for all upload collections, its essentially a shortcut to adding it yourself and I've just updated it so that you don't need to do that unless you really want to. Most people should only need to provide the adapter as the parameter. Otherwise 0 config.

I'm actually moving away from the virtual field and instead adding the afterRead hook to the collection itself due to the sizes param.

Also I really think an S3 plugin should keep things simple and shouldn't have to deal with resizing, deleting file, etc. so what's why I went with mine.

Completely valid, although I do think delete is something that should be supported.

I would rather not have to deal with resize and just upload the already resized files but that's currently not an option because:

1. People like yourself have expressed interest in not storing files locally.
2. I could pull the already resized images off the file system but that would require local storage.
3. I would like to make my plugin independent of local storage.

As I said I've pointed this out to the devs, I don't think it'd be a huge amount of work to pass the resized buffer along with the request but it's hard to do it well. In the mean time I'm resorting to copying the logic from payload itself.

In my case pointing to the full size image for adminThumbnail is very slow in the admin panel so I'd like to sort that out.

Complicated situation. Though I will sing Payload's praises for allowing me to fix it myself without restriction! 🥰

[jmikrut]

Hey @richardvanbergen and @jeanbmar - today was quite a productive day 😈

We just launched 3 new features which directly impacts this discussion:

• feat: allow completely disabling local file storage #272 - allow completely disabling local file storage
• feat: exposes auto-sized uploads on payload req #270 - expose resized file buffers to Payload req
• feat: allows upload resizing to maintain aspect ratio #271 - allow null or undefined as values for upload size width or height, which allows media to maintain their aspect ratio

💥

All of this is going to be released in 0.9.2 shortly.

@jeanbmar I need to go through this thread yet with a fine-toothed comb to catch up and understand your plugin. I am writing this in advance of fully reading the above thread but in any case, keep it up!

@richardvanbergen - I also need to find some time to go through your plugin again now that you've made some updates since last time I looked!

Very exciting day.......SUPER happy to see this type of dialog happening. You guys are awesome.

[jeanbmar]

Excellent. Disabling local storage will help a lot.

[richardvanbergen]

Wow you guys have had a good day! I wasn't expecting these issues to be addressed so soon!

These are all great for me, I didn't like the way I was having to support the resize myself and I can tidy up a lot of the feat/sizes branch that I was going to push now.

[jeanbmar]

I've updated the plugin to ensure safe filenames with database checks as a replacement for local fs checks

[quornik]

Hi Guys,
I've been using @jeanbmar solution for building upload and it works great, but I've stumbled upon a problem with serving images thru rest: Imaages uploaded to s3 are still served via PayloadCMS domain, instead of the s3 domain - the thumbnail thou is being served correctly. How can I change the url parameter of the uploaded images?

[afifaniks]

@afzaalahmad even after I set the url as you explained, it does not appear on my database. Although I can see it on the log.

This is my collection.

const Media = {
    slug: 'media',
    upload: true,
    fields: [
        {
            name: 'alt',
            label: 'Alt Text',
            type: 'text',
        },
        {
            name: 'url',
            type: 'text',
        },
    ],
    hooks: {
        afterRead: [
            ({ doc }) => {
                doc.url = `${process.env.CONTAINER_URL}/${doc.filename}`
                console.log(doc);
                // return doc; Doesn't work with or without return
            }
        ],
    }
}

export default Media;

Any thought?

Log

{
  id: '62ff7e3d16aef42a46e01955',
  filename: 'WhatsApp Image 2022-06-12 at 11.49.20 PM (1).jpeg',
  mimeType: 'image/jpeg',
  filesize: 108223,
  createdAt: '2022-08-19T12:12:45.717Z',
  updatedAt: '2022-08-19T12:12:45.717Z',
  url: 'https://omarastest.blob.core.windows.net/container/WhatsApp Image 2022-06-12 at 11.49.20 PM (1).jpeg'
}

[afzaalahmad]

So, you want this info saved in the db?
Which plugin are you using? and which service provider?

[afifaniks]

Using this one. '@payloadcms/plugin-cloud-storage/azure'

[afzaalahmad]

@afifaniks try adding this in the fields of your model:

{
      name: 'url',
      type: 'text',
      access: {
        create: () => false,
      },
      admin: {
        disabled: true,
      },
      hooks: {
        beforeChange: [
          ({ data }) => `${process.env.CONTAINER_URL}/${doc.filename}`
        ],
      },
}

I've tested it, and it'll solve your problem.

[afifaniks]

@afzaalahmad Thank you very much for your effort!

However, it looks like there is no reflection on the database.

The most interesting part is when change url to imgUrl it starts working!