Hub improvement checklist

[edogrigqv2]

This is a list of points that needs to be improved to consider it a polished gem.

• Storage - Zarr uses python's MutableMapping interface to interact with the filesystem. gcsfs and s3fs are part of fsspec and provide such an interface. However, those libraries are slow and we need to replace them with manually written ones. Aside from MutableMapping functionality we also need ls, rm, mv, cd, exists and other basic fs functionality as well. So by getting the url and credentials and input we should output MutableMapping with all of those functionalities. Right now there are 2 interfaces AbstractFileSystem and MutableMapping. These have been forced by fsspec. Either we should start contributing to them and improve their performance to our needs or we should go completely our way and combine those 2 interfaces into 1 as we don't need 2. The reason s3fs and gcsfs are slow is that the contributors were lazy to optimize the code properly. With proper implementation, those 2 libraries can to optimal speeds and we won't need to keep our own version. There is also optional listdir and rmdir functionality for zarr. Those should be added as well
• Dataset class is becoming a huge Blob. We need to decouple it into smaller parts. It can be decoupled by API vs Implementation and it can decoupled with get storage class (fsspec class) vs using fsspec class in the implementation of dataset.
  This 2-way decoupling should make it easier to use Dataset class. Next we need to get rid of class constructor from API, using function like hub.dataset(...) is much better than hub.Dataset. In the latter, the return type is fixed, while with API one we can be flexible. There is also a lot of optional functionality like to_pytorch or to_tensorflow IMPLEMENTED inside this class. We need to export implementations into separate "optional" files, that way it will be easier to manage everything.
• Better modules - right now our file separation is a mess. We should be consistent and document how we split our code into files. For example, a single class per file policy would do nicely. Other than that we can strictly define modules and what type of classes should be put into them. The way we are using modularity right now does not make a lot of sense.
• Better test coverage - code should be implemented with test coverage in mind. Functions should be designed to be easily unit tested.
• Caching - right now we are using caching around files. But after we get bytes from files it passes to decompression of Zarr. Meaning that the chunk will get compressed/decompressed over and over for every read/write. This issue can be eliminated if we had to cache one layer above (when we get NumPy arrays).
• Dataset download/upload feature that allows to download and upload datasets between local fs and cloud.
• Our code regarding user-inputted slice preprocessing is too complicated
• Add falke8 linter checks to CircleCI
• Add local pre-commit hooks for flake8 and black. This will reduce the number of false pushes to Github. Also adding pre-commit testing with coverage involved might be a good idea to remind contributors about coverage
• Smarter testing - We have 4 storage destinations. It would be nice that each dataset test could be tested on every destination. This is how it is done in Zarr.
• Type hinting - user API can accept a wide range of data types as input, but we should validate and normalize it down into consistent types and use type-hinting and assertions to emphasize that. This will improve auto-complete and will allow us to do fewer bugs. All arguments must be type-hinted and each type-hint should have a corresponding assertion.
• Add contributors/developers introductory on how to get started on contributing to the hub. The easier it is to get started, the more contributors we will attract.
• Splitting zarr tensor access from slice preprocessing. Currently, we have enormous logic how preprocess and postprocess slices and sliced data. Those can be moved into separate module other that getitem, setitem data in DynamicTensor. This will make those parts of our code more manageable.
• Security Warning: During today's CoDev we discovered that we are not taking care of security. Credentials are pickled by Ray and send over the network to different clusters, this is unacceptable and we should find smart ways to handle credentials without picking and sending them over the place. This should be handled in the future.

[edogrigqv2]

@davidbuniat @mynameisvinn Would like to hear your opinions regarding these points here. New suggestions would be nice as well.

[mynameisvinn]

Hey @edogrigqv2 you brought up lots of good points.

• Dataset. Agreed that we should tidy up Dataset and factor out methods like to_pytorch (eg subclassing Dataset). I'm organizing a community effort around Text APIs (eg higher level than to_pytorch, perhaps something like to_transformer) so it is a good opportunity to refactor.
• Caching. That's an interesting angle. How would that impact the partial file caching mechanism that's in place?
• Dataset download/upload. We should define an abstract base class so folks could create dataset helpers that are reproducible and consistent. I'll think about this and get back to you.
• Intro docs. Most of the members are not just new to Hub, they are new to open source projects in general. I'm not quite sure how to address this issue besides pointing them to resources.

[edogrigqv2]

Now we store cache in memory. We can add how many layers of caches we want, one layer is not affecting the other.