node-gyp support?

[toyobayashi]

It would be nice if node-gyp can work with emnapi on win/linux/mac.

[RReverser]

Yup, I'm planning to upstream the mentioned common.gypi that I intentionally kept separate from sharp-specific variables.

[toyobayashi]

Thank you.

I copied your common.gypi in gyp branch to node_modules/@tybys/emnapi, and ran

emmake node-gyp rebuild --nodedir=node_modules/@tybys/emnapi

Success on WSL2, but failed on Windows.

make: node-gyp rebuild --nodedir=node_modules/@tybys/emnapi
gyp info it worked if it ends with ok
gyp info using node-gyp@9.3.1
gyp info using node@16.15.0 | win32 | x64
gyp info find Python using Python version 3.9.7 found at "C:\Users\toyobayashi\AppData\Local\Programs\Python\Python39\python.exe"
gyp info find VS using VS2022 (17.2.32516.85) found at:
gyp info find VS "C:\Program Files\Microsoft Visual Studio\2022\Community"
gyp info find VS run with --verbose for detailed information
gyp WARN read config.gypi ENOENT: no such file or directory, open 'C:\Users\toyobayashi\Projects\emgyp\node_modules\@tybys\emnapi\include\node\config.gypi'      
gyp info spawn C:\Users\toyobayashi\AppData\Local\Programs\Python\Python39\python.exe
gyp info spawn args [
gyp info spawn args   'C:\\Users\\toyobayashi\\app\\nvm\\v16.15.0\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emgyp\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\app\\nvm\\v16.15.0\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emgyp\\node_modules\\@tybys\\emnapi\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=node_modules/@tybys/emnapi',
gyp info spawn args   '-Dnode_gyp_dir=C:\\Users\\toyobayashi\\app\\nvm\\v16.15.0\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=node_modules\\\\@tybys\\\\emnapi\\\\$(Configuration)\\\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\Users\\toyobayashi\\Projects\\emgyp',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\Users\\toyobayashi\\Projects\\emgyp\\build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
Traceback (most recent call last):
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\gyp_main.py", line 45, in <module>
    sys.exit(gyp.script_main())
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 687, in script_main
    return main(sys.argv[1:])
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 679, in main
    return gyp_main(args)
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 664, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2134, in GenerateOutput
    project_objects = _CreateProjectObjects(
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1936, in _CreateProjectObjects
    proj_path, fixpath_prefix = _GetPathOfProject(
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1883, in _GetPathOfProject
    default_config = _GetDefaultConfiguration(spec)
  File "C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 963, in _GetDefaultConfiguration
    return spec["configurations"][spec["default_configuration"]]
KeyError: 'Release'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Users\toyobayashi\app\nvm\v16.15.0\node_modules\node-gyp\lib\configure.js:325:16)
gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Windows_NT 10.0.22621
gyp ERR! command "C:\\Users\\toyobayashi\\app\\nodejs\\node.exe" "C:\\Users\\toyobayashi\\app\\nodejs\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--nodedir=node_modules/@tybys/emnapi"
gyp ERR! cwd C:\Users\toyobayashi\Projects\emgyp
gyp ERR! node -v v16.15.0
gyp ERR! node-gyp -v v9.3.1
gyp ERR! not ok
emmake: error: 'node-gyp rebuild --nodedir=node_modules/@tybys/emnapi' failed (returned 1)

It seems that gyp still uses msvs generator.

Another point, according to gyp doc

On OS X, linker settings are set via xcode_settings, on Windows via msvs_settings.

the ldflags won't work on win and mac.

So how can I use it on non-linux?

[RReverser]

It's because it still needs a bit more work for an actual PR, otherwise I'd already upstream it :P

[toyobayashi]

I tried node-gyp ... -- -f make on Windows, and it generated wrong path seperator \ instead of / in makefile, then emmake.bat make in mingw64 bash failed. On macOS, it generated arm64 related compile flags, which is complained by emcc. Out of curiosity, could you give me some hint how to workaround such gyp default behavior on different OS? Is it require upstream gyp code change, or all these problem can be solved by configuration file and some command line flags?

[RReverser]

could you give me some hint how to workaround such gyp default behavior on different OS

I haven't had time to look into that yet; I only used it via WSL for now but my focus has been elsewhere, sorry.

If you want to look into it, you're welcome to do so, I just 1) didn't have time but 2) want to make sure the authorship of common.gypi is preserved :)

[toyobayashi]

Oh sorry, I don't mean to rush you. Looking forward to your PR in patient, not worry

[RReverser]

No problem, just saying it's been messy and don't know when I'll get around to this. If you want, I can commit just the common.gypi for now so that at least Linux users would have a way forward, and we can figure out the other OS later.

[toyobayashi]

depends on your opinion. emnapi has few users, perhaps you are the first one. I don't mind to wait :)

[mmomtchev]

I am also interested in this feature and I am ready to contribute if anything is still missing. FYI, I am the author of SWIG Node-API and I planning to use emnapi to support transparent rebuilding for browser targets. I plan to to treat wasm as a separate, fourth node-gyp platform.

[toyobayashi]

@mmomtchev Thank you so much! I still don't know how to make node-gyp works for all platforms. I created a reproduce repo here: https://github.com/toyobayashi/emnapi-node-gyp-test

[mmomtchev]

Thanks, will give it a try. This project is a wonderful idea, it opens many new doors. I am also the author/maintainer of a number of Node.js addons.

[toyobayashi]

Good news! Found ways to build successfully on Windows and macOS.

nodejs/node-gyp#2823 (comment).

I have pushed to emnapi-node-gyp-test.

[mmomtchev]

One of the ugliest parts of the node-gyp build is that node-gyp insists on using relative filenames that recreate the source tree in build - which means that if you are referencing source files out of this tree - especially if they come from /usr/lib/node_modules or any other global installation - then your .o files tend to pop up in rather unexpected places - such as few directories above the build directory.

[toyobayashi]

common.gypi is added in https://github.com/toyobayashi/emnapi/releases/tag/v1.0.0.

upstream changes required:

• fix: support cross compiling for wasm with make generator nodejs/gyp-next#222
• Support rebuild and build when configure with -- -f make on Windows nodejs/node-gyp#2974

test with patches: https://github.com/toyobayashi/emnapi-node-gyp-test/actions

I think we can close this issue now since works in this repo has been done. Open issues at any time if have more specific problems.

[RReverser]

Awesome! I suspected that it would require upstream fixes but didn't want to dig into node-gyp itself 😅 Thanks for doing this (and for preserving the author notice)!