Javascript / Module support

[aheissenberger]

This works with bash helper (code at end of this note):

import {readFile} from "node:fs/promises"
const content = await readFile('test.txt',{encoding:'utf8'})

but when I add the functions to import/export the Data it fails:

import {readFile} from "node:fs/promises"
let transform = DM_getPanel(0);
const content = await readFile('test.txt',{encoding:'utf8'})

it fails with require is not defined in ES module scope - I think you use require in your wrapper or macro:

Error: [INFO] 2022-02-24T09:57:49 DataStation Runner (Go) development
[INFO] 2022-02-24T09:57:49 Evaling program panel
(node:4156) ExperimentalWarning: Fetch is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:4156) ExperimentalWarning: --experimental-loader is an experimental feature. This feature could change at any time
file:///private/var/folders/nq/kqg00wpj0yqcdzpzgts7ww440000gn/T/program-panel-792598270:7
  const fs = require('fs');
             ^

ReferenceError: require is not defined in ES module scope, you can use import instead
    at DM_getPanel (file:///private/var/folders/nq/kqg00wpj0yqcdzpzgts7ww440000gn/T/program-panel-792598270:7:14)
    at file:///private/var/folders/nq/kqg00wpj0yqcdzpzgts7ww440000gn/T/program-panel-792598270:30:17
    at ModuleJob.run (node:internal/modules/esm/module_job:197:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:341:24)
    at async loadESM (node:internal/process/esm_loader:88:5)
    at async handleMainPromise (node:internal/modules/run_main:61:12)

Node.js v17.6.0
[INFO] 2022-02-24T09:57:49 Failed to eval: exit status 1

    at evalInSubprocess (/Applications/DataStation Community Edition.app/Contents/Resources/app.asar/desktop/panel/eval.ts:190:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at IpcMainImpl.<anonymous> (/Applications/DataStation Community Edition.app/Contents/Resources/app.asar/desktop/rpc.ts:83:9)

1.) I suggest to implement it like an extra language or if possible as an extra option in the settings file (type: module) as it will require a new wrapper.
2.) wouldn't it be better to provide the functions (e.g. DM_getPanel) in a module which needs to be imported e.g. import {DM_getPanel} from 'datastation'

I used this bash script with experimental-loader to forces node to load file as module:

#!/usr/bin/env bash
set -e
exec /opt/homebrew/bin/node --experimental-loader 'data:text/javascript,let%20e=true;export%20function%20load(t,o,r)%7Bif(e)%7Bo.format=%22module%22;e=false%7Dreturn%20r(t,o,r)%7D' --experimental-fetch "$@"

[aheissenberger]

Ok - I understand the problem - you check for DM_ and then you only add the functions before the code:

function DM_getPanelFile(i) {
    return '/XXXX/DataStationProjects/.my-projec.results'+{"0":"a73a309d-b7b7-47eb-a8b4-f56404c36be2","1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #2":"a73a309d-b7b7-47eb-a8b4-f56404c36be2"}[i];
  }
  
  function DM_getPanel(i) {
    const fs = require('fs');
    return JSON.parse(fs.readFileSync('/XXXX/DataStationProjects/.my-projec.results'+{"0":"a73a309d-b7b7-47eb-a8b4-f56404c36be2","1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #2":"a73a309d-b7b7-47eb-a8b4-f56404c36be2"}[i]));
  }
  
  function DM_setPanel(v) {
    const fs = require('fs');
    const fd = fs.openSync('/XXXX/DataStationProjects/.my-projec.results066250cd-0972-4fad-86af-3f6bf91be317', 'w');
    if (Array.isArray(v)) {
      fs.writeSync(fd, '[');
      for (let i = 0; i < v.length; i++) {
        const row = v[i];
        let rowJSON = JSON.stringify(row);
        if (i < v.length - 1) {
          rowJSON += ',';
        }
        fs.writeSync(fd, rowJSON);
      }
      fs.writeSync(fd, ']');
    } else {
      fs.writeSync(fd, JSON.stringify(v));
    }
  }
  import {readFile} from "node:fs/promises"
  let transform = DM_getPanel(0);
  const content = await readFile('/XXXX/test.txt',{encoding:'utf8'})

I suggest to change your approach:

1. only add the import statement before the user code:
   CommonJS: const fs = require('fs'); ES modules: import fs from 'node:fs'
2. add the code for the function after the user code

Streaming could be improved by switching from your current JSON format to JSON Streaming.

import fs from 'node:fs'
/* BEGIN: user code */
 import {readFile} from "node:fs/promises"
  let transform = DM_getPanel(0);
  const content = await readFile('/XXXX/test.txt',{encoding:'utf8'})
/* END: user code */
function DM_getPanelFile(i) {
    return '/XXXX/DataStationProjects/.my-projec.results'+{"0":"a73a309d-b7b7-47eb-a8b4-f56404c36be2","1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #2":"a73a309d-b7b7-47eb-a8b4-f56404c36be2"}[i];
  }
  
  function DM_getPanel(i) {
    const fs = require('fs');
    return JSON.parse(fs.readFileSync('/XXXX/DataStationProjects/.my-projec.results'+{"0":"a73a309d-b7b7-47eb-a8b4-f56404c36be2","1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #1":"066250cd-0972-4fad-86af-3f6bf91be317","Untitled panel #2":"a73a309d-b7b7-47eb-a8b4-f56404c36be2"}[i]));
  }
  
  function DM_setPanel(v) {
    const fs = require('fs');
    const fd = fs.openSync('/XXXX/DataStationProjects/.my-projec.results066250cd-0972-4fad-86af-3f6bf91be317', 'w');
    if (Array.isArray(v)) {
      fs.writeSync(fd, '[');
      for (let i = 0; i < v.length; i++) {
        const row = v[i];
        let rowJSON = JSON.stringify(row);
        if (i < v.length - 1) {
          rowJSON += ',';
        }
        fs.writeSync(fd, rowJSON);
      }
      fs.writeSync(fd, ']');
    } else {
      fs.writeSync(fd, JSON.stringify(v));
    }
  }

[eatonphil]

You can't hardcode those DM_* functions by the way. They are generated with information for the specific function project/page/panel.

Yes I'd like to switch away from the current JSON format on disk.

only add the import statement before the user code:
CommonJS: const fs = require('fs'); ES modules: import fs from 'node:fs'

The reason I wanted to put the imports inside the functions is so that they couldn't possibly conflict with user code. If the user made their own variable fs the code would crash and they wouldn't know why.

add the code for the function after the user code

Why does it make a difference if it's before or after? Some other languages wouldn't allow this too since most languages require things to be declared before use.

[aheissenberger]

You can place this functions in a library and provide the variable parts e.g. command line parameter, or ENV variable. I would write them to a temp file and provide the path to the temp file as a parameter. Node and many other runtimes support preloading of libraries.

I moved the function after the user script to make it easy to add other imports without getting a conflict as imports need to be before any other code.

I have been tinkering with a simple function which will parse a NDJSON Format (or something where each object is in in one line to a stream. This allows to use a simple TransferStream to handle huge amounts of data without using a lot of memory.