Feature aws s3

[kaffein]

hey guys,

I just wanted to let you know that I have (at last) made some progress on the aws s3 command. Sorry for the little delay, I have been on another planet lately :).

I have made a minor modification to the early aws-invoke function for calling aws service APIs by adding an aws-client parameter. This parameter should allow any aws service clients (iam, s3 etc.) to be plugged in and should ease any upcoming integration.

At the moment, it covers (almost) all the features we had in the previous version of the s3 command. This is still a work in progress as I intend to add some (of the most useful) commands to it but feel free to let me know if there is anything that you would like me to put in.

I will keep you in the loop ;-)
thanks in advance

[devth]

Awesome! Glad to see a PR from you @kaffein 😃welcome back. Left a few thoughts.

[devth]

Can we make Availability Zones configurable?

[kaffein]

Sure, I will include it in the configuration 👍

[devth]

Same - more data?

[kaffein]

For the bucket list, we have more infos related to the buckets. I will paste the response format here and we will decide which data we will include in the formatted response.

[kaffein]

We have the following spec for the :ListBuckets API call :

:ListBuckets {:name "ListBuckets",
               :documentation "<p>Returns a list of all buckets owned by the authenticated sender of the request.</p>",
               :documentationUrl "http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTServiceGET.html",
               :request nil,
               :required nil,
               :response {:Buckets [:seq-of {:Name string, :CreationDate timestamp}],
                          :Owner {:DisplayName string, :ID string}}},

Apart from the Buckets list, we have an additional :Owner field included.
Do you want it to be included within the response @devth ?

[devth]

Sure. Can't hurt to have extra data behind the pipe :)

[devth]

Same - more data?

[kaffein]

Same for the objects listing, we have more infos returned by the API. I will paste the format here later and we will decide which infos we will keep in the formatted response.

[kaffein]

for the :ListObjectsV2 API call, we have the following spec :

:ListObjectsV2 {:name "ListObjectsV2",
                 :documentation "<p>Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. Note: ListObjectsV2 is the revised List Objects API and we recommend you use this revised API for new application development.</p>",
                 :request {:Prefix string,
                           :StartAfter string,
                           :Bucket string,
                           :EncodingType string,
                           :Delimiter string,
                           :FetchOwner boolean,
                           :RequestPayer string,
                           :ContinuationToken string,
                           :MaxKeys integer},
                 :required [:Bucket],
                 :response {:Prefix string,
                            :StartAfter string,
                            :EncodingType string,
                            :Delimiter string,
                            :NextContinuationToken string,
                            :CommonPrefixes [:seq-of {:Prefix string}],
                            :ContinuationToken string,
                            :Contents [:seq-of
                                       {:Key string,
                                        :LastModified timestamp,
                                        :ETag string,
                                        :Size integer,
                                        :StorageClass string,
                                        :Owner {:DisplayName string, :ID string}}],
                            :MaxKeys integer,
                            :IsTruncated boolean,
                            :Name string,
                            :KeyCount integer}},

I only took the :Contents but the :response key is assoc-ed to much more information, mostly meta-data such as pagination-related stuff (I will also have to think about how to manage it too by the way ^^).
Which other keys do you think we should include in the response for this one ?

[devth]

I think everything. I'm aiming to make data behind pipes super powerful. I'd prob only omit something if it was massive, unrelated, or a security issue.

[kaffein]

Awesome! Glad to see a PR from you @kaffein welcome back. Left a few thoughts.

thanks a lot @devth :) ... and thanks for the feedback!! I will check them out

[kaffein]

hey guys,

Sorry for the delay on this PR, I was a little bit busy the last few months ^^ but I think we are ready this time.

So this PR emulates a few aws s3 CLI commands :

• bucket creation
• bucket listing
• bucket content listing
• object copying
• object deletion
• (empty) bucket deletion

There were a few interesting commands in the official aws CLI that were not available through the API though, for e.g : renaming/moving (mv) objects, or sync-ing buckets content (à la rsync). Especially for the renaming/moving objects, I wonder if it is possible to have a combination of copy and delete (in an atomic operation) to emulate it.

I can make another round of testing to make sure we do not have regressions and I will include the screenshots here if you want @devth. It will probably ease your job but of course feel free to play with the PR to see if there are things that need to be fixed.

thanks in advance

[kaffein]

Yo,

just to give a quick update on the last round of tests for the aws s3 feature. I have included some screenshots so that we can have an idea of how it looks like. The whole thing can probably be improved so feel free to suggest anything that can help us make it better ;-).

• bucket creation

• bucket creation with an already existing bucket name

• bucket listing

• bucket content listing

• Non existent bucket content listing

• object copying

• invalid object copying attempt

• object deletion

• (empty-only) bucket deletion

can you PTAL @devth

thanks in advance ^^

[devth]

@kaffein looks awesome! I will try to find some time to review this tomorrow if that's ok.

[kaffein]

Take your time ... we're not in a hurry 😉😊

Thanks 🙏

[devth]

Playing with this now!

[kaffein]

Enjoy and take your time!! it's a big chunk haha

[devth]

This is awesome! I setup an AWS account and played with it. I didn't test every command yet. Found a couple things and left some comments.

Thanks! ☁️ ☁️ ☁️

[kaffein]

I might want to refactor the whole aws suite of commands : there are probably some possibilities to generate all the needed functions from the ops (operations) made available by each aws client.

[devth]

@kaffein code gen sounds cool and makes sense for such a large set of commands.