Current webpack configuration leads to memory issues when building LORIS

[MaximeMulder]

Issue

Describe the bug

We have had recurrent memory issues when building LORIS on our MCIN VMs, and often had to ask IT for more memory. Recently, many people were unable to build my very cool and very mysterious PR #9190, which I also did not fully build myself as I only built the module that uses it during development.

After a little investigation, it seems the root cause of our issues is that the code located in <loris-root>/jsx/ is copied in each LORIS module that imports it, which explains why my PR in particular caused memory issues while others, which only touched a specific module, did not.

The detailed results of my investigation, is that if you look at these files (for instance) :

• modules/dicom_archive/js/dicom_archive.js.map
• modules/datadict/js/dataDictIndex.js.map
• modules/imaging_browser/js/imagingBrowserIndex.js.map
  which are genereated by webpack during compilation (and thus not in the git repository), and look for case 'time':\n (ctrl+F), which is from jsx/Filter.js, you can see that it is copied, along with its file and many others, in each module that transitively imports it (these specific modules all use a filterable data table), leading to code duplication and thus huge memory consumption.

Solution

The solution is obviously to change our webpack configuration so that the common Javascript code located in jsx/ is only inluded once for the whole project instead of being copied in each module. I have not done the research yet on how we should do that.

[driusan]

I don't have any expertise but I think the place to start if you're looking into that is probably here: https://webpack.js.org/guides/code-splitting/

[MaximeMulder]

Okay, so I looked at our webpack .configuration and made some changes : Even without using code splitting (which while having the same amount of duplication in the client code than we did before), it is easy to have enormous performance gains while building LORIS. In our current approach, we build one full project for each module. By tweaking a few things, I managed to fully build LORIS on my VM in less than 45s without issue. If we remove the duplication with code splittng, I suspect we can get even more gains.

I would very much like to to work on code splitting tomorrow, along with a few other improvements in our configuration.

[MaximeMulder]

Okay, so after talking with @driusan, I will write a PR for the following changes:

• Change the webpack configuration so that it uses a single configuration object with many entries instead of one configuration object per module. Projects can currently add other configuration objects so I will also retain this feature.
• I will also take this opportunity to upgrate the webpack.config.js file to Typescript.

These changes are minimal but they significantly speed up the LORIS building process and solved the memory issues on my VM. They also are not breaking any current code, which is nice since we probably don't want to get unnoticed import errors again so close to a release. I will see if I can add de-duplication into the mix, but if I cannot do so in a satisfactory way and without breaking current code, I think it is best post-poned to a larger refactor.

For a larger webpack refactor, I also have further changes in mind, which I may eventually argue for in a separate issue. But as I said, my objective for now is to not break the current code.

If you have any thoughts @ridz1208 @laemtl, I'd be glad to hear them. If everything goes well, I may get the PR opened today.

[ridz1208]

@MaximeMulder on the project level there is usually a webpack-project.config.js file (see the CBIG repo for example) can you make sure to detail what would need to change in that file to remain functional ?