Feature: "List" api

[jrochkind]

Ability to do some kind of ls-like behavior in storage-agnostic way on a storage.

Use cases:

• Iterate through entire storage contents, to find "orphans" files not currently being pointed to by a model (perhaps left around by a bug or backup recovery). In storage-agnostic way. (Maybe you want to do this on just a portion of a very large storage using a prefix, instead of on the whole storage at once?)

• Provide a UI to let user navigate/list contents of a storage, in storage agnostic way, for various purposes, such as selecting a file.

API suggestion

Storage#list yields/returns string paths

We can have that Storage#list method, which would yield file paths. (May also take a prefix to list at just a certain "directory")

some_storage.list do |path|
    path # => "some/path/file.jpg"
end

some_storage.list(prefix: "some/path/") { |path| ... }

"Directories" would not be yielded (because they don't really exist as "things" on some storage systems), only individual files.

Should there be a way to get an array returned instead of using a block arg? Or actually, if we use the Enumerable stuff if no block is given, to support chaining, we can get that and more I think? So that should be encouraged/required in API?

Shrine.list yields/returns UploadedFile objects

Then we could have a Shrine#list/Shrine.list method which would wrap this into an enumerable of Shrine::UploadedFile objects. An UploadedFile object is our general abstraction for storage-located files, already letting us do various things with it like get a url or delete.

Shrine.list(:store) { |uploaded_file| ... }
Shrine.list(:store, prefix: "some/path/") { |uploaded_file| ... }

an_uploader.list { |uploaded_file| ... }
an_uploader.list(prefix: "some/path") { |uploaded_file| }

Similar to above, should probably return Enumerator if no block given, to support chaining and other useful things.

Implementation sketch

class Shrine::Storage::S3
  def list
    bucket.objects(prefix: prefix).each do |object|
      yield object.key[/^(#{prefix}\/)?(.+)/, 2]
    end
  end
end

class Shrine::Storage::FileSystem
  # we already have this as #list_files
  def list
    Pathname("#{directory}/") # add trailing slash to make it work with symlinks
      .find
      .each { |path| path.relative_path_from(directory).to_s if path.file? }
  end
end

module Shrine
  module Plugins
    module List
      module ClassMethods
        def list(storage_key, &block)
          new(storage_key).list(&block)
        end
      end

      module InstanceMethods
        def list
          return enum_for(__method__) unless block_given?

          storage.list do |id|
            yield self.class::UploadedFile.new(id: id, storage_key: storage_key)
          end
        end
      end
    end

    register_plugin(:list, List)
  end
end

This is "Optional" Storage API

All storages are not required to implement list, as shrine itself does not actually use it for basic functionality.

But it should be encouraged, to allow storage-agnostic listing.

The Linter should test if if the relevant method is present, but allow it not to be present.

When testing, Linter makes sure prefix arg works, and makes sure if no block is given you get an Enumerator/Enumerable returned that behaves appropriately.

(Possible future Linter "maximal mode" where it requires this and various other non-mandatory but recommended features, so you can see if a given storage lints under 'maximal mode' with all recommended features).