Proposal: Fundamental changes to django-filer

[jrief]

django-filer is a fundamental part of the django-CMS ecosystem and even used outside of it. Its origins date back until 2009 and one does not have to be a Django expert to recognize its age. I was part of this project since 2013 and contributed a few features, for instance high resolution support for retina displays, SVG and WebP support. Lately I was asked by my team to improve the functionality for copying and moving files, but it turned out that this would be quite a difficult task. Reason is that unfortunately django-filer has not been maintained well in a future proof way. I therefore would like to list a few legacies, and explain why this makes it so hard to maintain.

Some of the JavaScript files have been taken from existing libraries, but unfortunately only the minimized part. If you look at these files

• https://github.com/django-cms/django-filer/blob/master/filer/static/filer/js/libs/class.min.js
• https://github.com/django-cms/django-filer/blob/master/filer/static/filer/js/libs/dropzone.min.js
• https://github.com/django-cms/django-filer/blob/master/filer/static/filer/js/libs/fileuploader.min.js
• https://github.com/django-cms/django-filer/blob/master/filer/static/filer/js/libs/mediator.min.js

it is hard to impossible to maintain them. The unminimized versions that are not part of the project, and their originating projects are gone, unmaintained themselves or untrackable. In the case of fileuploader.js and dropzone.js these projects probably are unnecessary, because modern browsers can handle this way better and out of the box now.

Django-filer's JavaScript code relies on jQuery version 1.11. Current Django version are shipped with jQuery 3.6. This means that we have to work with two different versions of jQuery and disable one in order to enable the other. This approach is quite error prone.

With this legacy, adding larger new features to django-filer became impossible, especially if they require code running in the browser.

But the server-side part also has some issues.

django-filer relies on django-mptt which states that it currently is unmaintained. But even if it would be maintained, the used algorithm is prone to race conditions and performs badly on writes if the tree gets too large. This is because all the nodes of the tree must be updated, whenever a node is inserted or deleted.

django-filer also relies on django-polymorphic. Polymorphism currently is only used to distinguish between "normal" files and images. I assume that there was an intention to support more types of files, but it turned out that adding an additional model for each file type makes everything quite cumbersome. Therefore nobody implemented a model for PDF or Videos.

django-filer created it own Permission model. This probably is due to the fact, that in 2009 there wasn't anything equivalent. However, nowadays I would use Django's internal Permission model and add a layer to apply it per object. Unfortunately django-guardian isn't an option, because it can't handle trees.

My proposal

Rewrite django-filer from scratch.

Use the Folder model as a starting point. Use django-tree-queries to represent the tree of folders. It uses CTEs and hence is much faster and safer than django-mptt.
For files, use the File model as a starting point. Remove django-polymorphic and instead use proxy models to represent different file types. This avoids the 1:1 relation between the inheriting and the base class. Use a JSONField on each File object to store Meta-Data. This means that for instance, image specific meta data such as width, length, focal point, etc. are stored in JSON rather than in relational columns. It also allows to store EXIF data and other arbitrary data. By using django-entanged it also is possible to edit data in that JSONField.

There will be a migration path.

For the Javascript part handling the tree, I would propose DnD Kit. It is written in pure Typescript without external dependencies. React is required as a third party library. File uploading through drag & drop can be done without library. This only takes a few lines of code.

I will also remove all the popups and views using iframes and replace them against a standard HTML Dialog.

As a builder and JS-compiler I made good experience with ESBuild. It can be started directly from npm and doesn't require any external builder such as Webpack, Grunt, Gulp or whatever. It can also handle SCSS. All extra client code will be written in TypeScript.

I believe that I can write such a library, but I would love to hear a commitment from the community, because a new version of django-filer only makes sense, if it is used as a replacement. I don't want to compete against the current version.

[marksweb]

This sounds like a positive start to me 👍

[fsbraun]

Sounds great! 👍

One thought that comes to my mind is: Do we need a tree library at all? Just have any Folder/File have a foreign key to a folder (or empty for the root folder). It would save us a dependency. What would the drawback be of such a minimal solution?

[jrief]

Model File has a foreign key to Folder, but Folder has a foreign key to itself. This causes a typical n+1 problem when performing queries over a deep hierarchy of folders, for instance looking for all descendants or ascendants.

[fsbraun]

Looking into the sql queries generated by filer, it seems that thumbnails muddy the waters. Tree issues are typically small. That's just a non-representative observation. Maybe we need to investigate this? If thumbnail calls to the db can be avoided using json fields or prefetches ...

[jrief]

Edit: Since we don't really need a lot of tree traversals, and sich @fsbraun already removed django-mptt, using a CTE-based library is presumably overengineered. I therefore would follow this approach a go with simple adjacent lists.

[yml]

I come to this discussion from the perspective of this ticket #1406.

This proposal has many good things, and I agree that the front-end representation could be modernized and improved. The tight coupling between filer and easy-thumbnail also can be problematic at some scale as mentioned above. (I say this more easily because I am the one that put it in :-) ) The "internal" beauty of django-filer is that it is easy to extend and change some of its core behavior entirely by swapping the model. This pattern has been made famous by django.contrib.auth with the User model. I am not going to articulate all the great side effects of the swappable User model or Image model in django-filer.

I understand you want to remove this option and replace it with a JsonField. On one hand I can see this be a solution for some project that needs to add a bag of metadata, but on the other end, I can see a few issues with this solution. It is a non-backward compatible change for many projects using django-filer. It limits a lot of the kinds of changes you can make. Here is a non-exhaustive list of things that you will not be able to do:

• Custom admin.py for images (formfield_overrides, fieldsets, autocomplete_fields, ...)
• FK to other content
• change the order
• some db optimization
• change the thumbnailer pipeline to use something like (https://www.thumbor.org/)

The intention is to use one JSON field and proxy models. This allows us to add custom behavior in a very simple way.
I always found it very painful having to work with swappable models. As far as I remember, Django introduced some helpers for this. Do you know of any project using that?

My own experience has been different. I had good experiences with swappable models in multiple contexts. It takes some leg work to get going, but the customization and flexibility you get from it are impressive. As you can tell be #1406 I would rather have more of them than less.

Here it is an example of django-allauth taking advantage of django.contrib.auth swappable User model.

[fsbraun]

@jrief @yml Thinking this through I come to the conclusion that this is not necessarily an either or. Why don't we do the following:

• Replace the File polymorphism with a JSON field made to contain meta information on the file depending on its type (such as size or copyright for an image). The Image model then is just another name for the File model.
• Not sure this is needed for Folder but would need to be discussed

plus

• Keep the Image model ( == File model) swappable to allow for further user-defined fields or methods operating on the existing JSON field.
• Make the Folder model swappable (how do we manage foreign keys to a swappable model?)

Finally, I'd suggest to modify the File model in such a way that easy-thumbnails becomes officially swappable, too. This would require a refactor to collect all dependencies on easy-thumbnails in the File model.

In this scenario we would lose the polymorphism of File and keep the swappable model architecture. What do you think?

[yml]

Replace the File polymorphism with a JSON field made to contain meta information on the file depending on its type (such as size or copyright for an image). The Image model then is just another name for the File model.

Copyright, licence, author are often represented as FK so not really well suited for json field. The good thing with swappable model is that if people want to use them they can.

Taking a step back could someone point me to the list of bug issue that would be solve by removing django-polymorphic.

[yml]

In this scenario we would lose the polymorphism of File and keep the swappable model architecture.

It is difficult for me to wrap my head around how we would have different detail admin pages for File, Image, Video with different fields exposed and different behavior without having one Model for each. I can somehow guess that we could replicate some of what is done by django-polymorphic with proxy models but I have doubt that it would be simpler, faster or more convenient.

[jrief]

@yml

Copyright, licence, author are often represented as FK so not really well suited for json field. The good thing with swappable model is that if people want to use them they can.

Instead of FILER_IMAGE_MODEL there will be a FILER_FILE_MODEL, so that one can swap out this model against his own implementation, although I believe that this will rarely be used. Adding a swappable folder makes the architecture too complicate, because we then have model File requiring a swappable foreign key pointing on that model.

Just for your information, in django-entangled you can store foreign keys in JSON.

It is difficult for me to wrap my head around how we would have different detail admin pages for File, Image, Video with different fields exposed and different behavior without having one Model for each.

The problem with the current model is, that although well intended, after more than 13 years of django-filer no one implemented anything else but "Image". The key to specialization is its mime_type. This field will determine which editor to use. Currently we have two fields to store the file-type, said mime_type and polymorphic_ctype_idand this thwarts the 2NF.

File system also do not implement polymorphism but you presumably agree that this doesn't prevent you from working with files in an operating system. I want implementers to be able to add editors in a manner as simple as for operating systems.

I'm using this pattern in djangocms-cascade since ~8 years with great success. There all Django-CMS plugins share the same model. The model type is stored in field CMSPlugin.plugin_type and determines which editor shall be used. All data is stored in a JSON field. By doing so, it turned out that adding or changing plugins to Django-CMS is way faster to implement, rather than using the classic approach of creating one model for each plugin type (and btw. keeps the number of tables in your database low).

By refactoring django-filer towards this simpler model, I hope to achieve the same effect there.

Please join us on our weekly meetings to discuss this in detail.