Add support for installing ESM module nodes

[hlovdal]

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Proposed changes

Currently node-red fails when attempting to install a node written as an ESM module, e.g.

19 Sep 18:59:53 - [info] Server now running at http://127.0.0.1:1880/
19 Sep 18:59:53 - [info] Starting flows
19 Sep 18:59:53 - [info] Started flows
19 Sep 19:01:05 - [info] Installing module: @hlovdal/node-red-lowercase-in-typescript, version: 1.0.0
19 Sep 19:01:12 - [info] Installed module: @hlovdal/node-red-lowercase-in-typescript
19 Sep 19:01:12 - [info] Added node types:
19 Sep 19:01:12 - [info]  - @hlovdal/node-red-lowercase-in-typescript:lower-case : Error [ERR_REQUIRE_ESM]: require() of ES Module .../.node-red/node_modules/@hlovdal/node-red-lowercase-in-typescript/src/lower-case.js from .../node-red/packages/node_modules/@node-red/registry/lib/loader.js not supported.
Instead change the require of lower-case.js in .../node-red/packages/node_modules/@node-red/registry/lib/loader.js to a dynamic import() which is available in all CommonJS modules.

This pull request fixes that by using dynamic import() instead of require.

Checklist

• I have read the contribution guidelines
• For non-bugfix PRs, I have discussed this change on the forum/slack team.
• I have run npm run test to verify the unit tests pass
• I have added suitable unit tests to cover the new/changed functionality

[hlovdal]

I am not sure why the EasyCLA status is not updated, it is fine in this pull request.

[hlovdal]

/easycla

[hlovdal]

TL;DR

The test failure that occurred on my CI build is not related to my changes, it is inherited from the existing code and applies to both dev and master.

Test results & conclusion

I have now run testing overnight and the results are:

$ wc -l success-dev failure-dev
 1312 success-dev
   86 failure-dev
 1398 total
$ wc -l success-installing_esm_nodes failure-installing_esm_nodes
 1297 success-installing_esm_nodes
  103 failure-installing_esm_nodes
 1400 total
$ wc -l success-master failure-master
  416 success-master
   37 failure-master
  453 total
$

Or in percentages, the test fails in 6.2%, 7.4% and 8.2% of the times on the various branches, and the total failure rate for all is 7.0%.

The value that fails is almost 200 which it is compared agains, over half is 199 (details in the last step).

So conclusion, the timing granularity/accuracy of this failing test is not good enough and it will fail from time to time (around 7% of the time).

Details

Step one was to limit testing to just the file that triggered the failure by modifying Gruntfile.js to have

            all: { src: ["test/nodes/core/function/10-function_spec.js"]},
            core: { src: ["test/nodes/core/function/10-function_spec.js"]},
            nodes: { src: ["test/nodes/core/function/10-function_spec.js"]}

Step two: run the tests through a script that records the success or failure from each test:

#!/bin/sh

touch success-$1 failure-$1
while true
do
        if npm run test
        then
                date >> success-$1
        else
                date >> failure-$1
        fi
        echo ====================================================
        wc -l success-$1 failure-$1
        echo ====================================================
done

Step three was to test on all three branches:

timeout 8h ./run.sh installing_esm_nodes; git checkout dev; timeout 8h ./run.sh dev; git checkout master; ./run.sh master

Step four, analyse failure values.

To be able to see what the failure values were I ran the testing inside a script session, capturing text output to a typescript file.

$ awk '/Message received before init comple/{ A[$9]++ } END {total=0; for (i in A) {total += A[i]} for (i in A) {printf "%d %3d %5.1f%%\n", i, A[i], 100*A[i]/total; }}' typescript 
197   1   0.4%
198  88  38.9%
199 121  53.5%
200  16   7.1%
$

(the typescript file includes a few more tests than those captured in the failure-* files for those of you double checking the numbers)

[hlovdal]

Same results with node.js v18.17.1:

  1260 node18-success-installing_esm_nodes
    73 node18-failure-installing_esm_nodes
  1244 node18-success-dev
    90 node18-failure-dev
  1241 node18-success-master
    91 node18-failure-master
  3999 total

[fbuys]

I am excited for this to be released, I left a comment if it helps.

[fbuys]

I wanted to share this approach in case it is useful, just an idea not sure if it is 100% correct:

    (async () => {
        const moduleNamespaceObject = await import(node.file);
        // CJS will always have default, ESM might have default. (details in note below)
        let r = moduleNamespaceObject.default ? moduleNamespaceObject.default : moduleNamespaceObject;
        // Babel related workaround. (references in note below)
        r = r.__esModule ? r.default : r;
        return createLoadPromise(node, r);
    })();