shebang: `#!/usr/bin/env node` — a `BOM` issue

[SMotaal]

⁉️

Folks, quick question about shebang on macOS!

Package: ~/blah/blah/package.json

{
   // …
   "type": "module",
   // …
}

Module: ~/blah/blah/blah.js

#!/usr/bin/env node

console.log(process);

Command: node blah.js # in ~/blah/blah

(node:19422) ExperimentalWarning: The ESM module loader is experimental.
file:///~/blah/blah/blah.js:1
#!/usr/bin/env node
 ^
SyntaxError: Invalid or unexpected token
    at Loader.moduleStrategy 
       (internal/modules/esm/translators.js:83:18)
    at async link
       (internal/modules/esm/module_job.js:37:21)
When did that change?

---

Checks:

❌ v13.x unflagged
❌ v12.14.1 flagged
✅ { type: 'commonjs' } — how this was passing is beyond me 😊
✅ blah.cjs

✅ "I'm not getting that error in node.js 13.6.0… maybe an issue in darwin? I'm not able to reproduce this issue with 13.6.0 on linux x64, installed by nvm." — @coreyfarrell

✅ without BOM (thanks @devsnek — it relates to a discrepancy when BOM is present).

---

Module: ~/blah/blah/blah.js

- #!/usr/bin/env node
+ // #!/usr/bin/env node

Command: node blah.js # in ~/blah/blah

process {
  version: 'v13.6.0',
  versions: {
    node: '13.6.0',
    v8: '7.9.317.25-node.26',
    uv: '1.34.0',
    zlib: '1.2.11',
    brotli: '1.0.7',
    ares: '1.15.0',
    modules: '79',
    nghttp2: '1.40.0',
    napi: '5',
    llhttp: '2.0.1',
    openssl: '1.1.1d',
    cldr: '36.0',
    icu: '65.1',
    tz: '2019c',
    unicode: '12.1'
  },
  arch: 'x64',
  platform: 'darwin',
  release: {
    name: 'node',
    sourceUrl: 'https://nodejs.org/download/release/v13.6.0/node-v13.6.0.tar.gz',
    headersUrl: 'https://nodejs.org/download/release/v13.6.0/node-v13.6.0-headers.tar.gz'
  },
  // …
  config: {
    target_defaults: {
      cflags: [],
      default_configuration: 'Release',
      defines: [],
      include_dirs: [],
      libraries: []
    },
    // …
  },
  // …
  env: {
    // no NODE_… stuff
  },
  // …
}

[hybrist]

I assume the same thing happens if the file is called blah.mjs (removing the specifics of the package.json field from the repro)?

[hybrist]

One quick meta note is that V8 should support the shebang natively and IIRC our module implementation depends on that. So it feels really weird that it would be platform-dependent..?

[hybrist]

Tried on my mac (version installed via nvm, so official release build) with both .js and .mjs. In both cases it seemed to work. Are there any other details I might have missed?

$ node test.js
(node:12797) ExperimentalWarning: The ESM module loader is experimental.
file:///tmp/shebang/test.js darwin
$ node -v
v13.6.0
$ cat test.js
#!/usr/bin/env node

import {platform} from 'os';

console.log(import.meta.url, platform());

[hybrist]

Ah, missed { type: 'commonjs' } (!). Wait - if type is CommonJS, why is it trying to run the source as a module?

P.S.: internal/modules/esm/translators.js:83 on v13.6.0 is the handler for files in module format which shouldn't apply to CJS source code.

[hybrist]

Still, same result after running as CJS (removing my bad type field from package.json):

$ cat test.js
#!/usr/bin/env node

const {platform} = require('os');

console.log(platform());
$ node test.js
darwin
$ node -v
v13.6.0

[GeoffreyBooth]

I haven't tried debugging this, but I remember that earlier versions of our loader included Node stripping out the shebang line; I think when V8 added support for it, Node's code was removed. So there might be a bug in V8 shebang handling.

[devsnek]

@SMotaal does your file start with a utf8 byte order mark?

echo \uFEFF#!test | v8
V8 version 8.1.117
d8> (d8):1: SyntaxError: Invalid or unexpected token
#!test
 ^
SyntaxError: Invalid or unexpected token

[SMotaal]

Yup, that does it :)

I default to this in vscode:

I saved as UTF-8 (no BOM) and it worked!

[ljharb]

Is it intended that a starting BOM would not be removed as part of the shebang?

[devsnek]

it shouldn't be removed as part of the hashbang, but i'm surprised that node doesn't strip utf8 bom when reading source files anymore.

[SMotaal]

I am inclined to say it is an OS problem:

bom.sh

#!/usr/bin/env sh

echo "executed!";

• with BOM:

  ./bom.sh: line 1: #!/usr/bin/env: No such file or directory
  executed!
  

• without BOM:

  executed!
  

[SMotaal]

which is more disturbing honestly :)