Build node-sqlite3 for node-webkit on Travis CI #252

Mithgol · 2014-02-06T11:55:00Z

Travis CI configuration changes proposed in #251.

fork node-sqlite3
try tweaking the .travis.yml to include a matrix for different node-webkit versions
download the right stuff
build
run node-pre-gyp package testpackage

Note: there's currently no API to determine the most recent version of node-webkit over the Web. And thus the desired node-webkit's version is given after Node's version in the matrix.

TODO:

build node-sqlite3 for 64bit node-webkit on Linux (because there are 64bit node-webkit builds for Linux, though there aren't for Mac/Win)

There's currently no API to determine the most recent version of node-webkit over the Web.

Note: `make clean` happens after `npm install nw-gyp -g` because we intend to use `nw-gyp` later to rebuild the existing build of node-sqlite3.

Release notes https://groups.google.com/d/msg/node-webkit/bPjur7aFaQc/4TP6o3SM-8sJ say that 0.8.4 fix improves the behaviour on OS X.

It seems (in installation logs) that nw-gyp does not contain any binary (i.e. compiled) addons. Thus it's fine if nw-gyp survives 64bit→32bit switch of our Node.js environment on Linux.

This commit does also prepend `on Linux 64 bit:` to some comments (making it easier to understand what subplatform is targeted). On Linux nw-gyp is installed earlier (in 64 bit mode) and thus we can refrain from its reinstallation for 32 bit, speeding things up.

Try a feature originally introduced in node-pre-gyp v0.4.2 commit mapbox/node-pre-gyp@370024a for mapbox/node-pre-gyp#14.

Working around a temporal inavailability of node-pre-gyp v0.4.2. See mapbox/node-pre-gyp#35 for details.

Otherwise the following check fails: https://github.com/springmeyer/node-pre-gyp/blob/3d4632a7333d420d6423299b143f5391b626e904/lib/util/versioning.js#L75-79

Mithgol · 2014-02-13T10:12:51Z

Line 1700 of a Travis CI log indicates that node-pre-gyp needs an improvement of its validation function.

Otherwise it runs node-webkit ./lib/sqlite3, where ./lib/sqlite3 is the value of main from package.json (which is, as far as I undersand, not a node-webkit's application) and where node-webkit is… Zalgo knows where that node-webkit came from; from Darwin, I suppose; but node-webkit's executable is nw.exe on Windows and nw on Linux.

Mithgol · 2014-02-13T10:53:12Z

I've opened a pull request (mapbox/node-pre-gyp#36) to add nw.exe on Windows and nw on Linux.

A pull request (mapbox/node-pre-gyp#36) is opened to land a partial fix for node-pre-gyp's validation process. This commit uses a source branch of that pull request. After this commit, node-pre-gyp's validator is expected to launch `nw` instead of `node-webkit` on Travis CI's Linux. A former ENOENT error is expected to be replaced by a node-webkit's error when the engine realises it's given a node-sqlite3 module's JavaScript instead of some HTML5 application.

Commit Mithgol/node-pre-gyp@eaea6e1 is now included in the node-pre-gyp's source used here.

Mithgol · 2014-02-13T12:10:43Z

The 6655d7c's resulting Travis CI log shows that, while mapbox/node-pre-gyp#36 seems okay (i.e. nw instead of node-webkit is called on Linux), two more obstacles are remaining:

node-pre-gyp cannot test node-sqlite3 on node-webkit because node-webkit's executable is missing (ENOENT on line 1702 of the log);
node-pre-gyp cannot test node-sqlite3 on node-webkit because, unlike Node.js, node-webkit does not run javascripts (such as ./lib/sqlite3).

The former of those two obstacles is to be solved by downloading and unpacking of the corresponding (i.e. platform-dependent) node-webkit engine. (This feature can be done in commits here, but should later be transferred to node-pre-gyp.)

The latter of those two obstacles is to be solved by adding some HTML5 tester to node-pre-gyp (a mere require of a module given in its command line would suffice).

Mithgol · 2014-02-13T12:24:18Z

The a86e300's resulting Travis CI log is worse than I imagined.

It seems that Linux 64bit node-webkit can't run on Travis (lines 1271—1273):

node-pre-gyp ERR! stack Error: Command failed:
node-pre-gyp ERR! stack (nw:2106): Gtk-WARNING **: cannot open display:
node-pre-gyp ERR! stack [2109:0100/000000:ERROR:zygote_linux.cc(478)] write: Broken pipe

Mithgol · 2014-02-13T12:32:25Z

Of course, there must be a way to trick Gtk into believing it has a display. But that's totally outside of my area of knowledge, at least currently.

xvfb recipe: http://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-GUI-(e.g.-a-Web-browser) sleep 3 recipe: http://docs.travis-ci.com/user/gui-and-headless-browsers/#Starting-a-Web-Server

Mithgol · 2014-02-13T13:06:19Z

Read nwjs/nw.js#769 and understood that it's possible to start xvfb using a couple of recipes from Travis CI docs.

Made a new commit to test if setting $DISPLAY is enough to trick node-webkit.

Mithgol · 2014-02-13T13:14:26Z

Got Package appears valid on this log, line 1272. It seems that xvfb is enough for a headless node-webkit.

Mithgol · 2014-02-13T15:03:48Z

Yet another unexpected obstacle:

nw: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

Why node-webkit for Linux 32 bit expects libX11.so.6 but 64 bit node-webkit does not?

More magic.

springmeyer · 2014-02-13T16:24:19Z

Great progress and really valuable learning. The testing of binaries is a hard problem in general let along on a headless server with a GUI browser. The most important thing to me is to have builds automated, so I'm not burdened with this when releasing a new version. The big gap here is windows. Want to put thought into that. Really wish Travis supported windows, what alternatives do we have?

This commit also erases `sudo apt-get update --fix-missing` introduced in one of the previous commits, because it didn't help.

Mithgol · 2014-02-13T16:48:24Z

First thing that comes to mind is that Travis CI admins at travis-ci/travis-ci#216 said “we hope to have some news to announce in the coming weeks” but it's been ≈7 months since that comment.

Mithgol · 2014-02-13T17:13:33Z

To make Travis CI build and test GEOS for Windows it was suggested to use Wine.

Follow the following suggestion: http://askubuntu.com/questions/363878/how-to-install-32-bit-matlab-in-ubuntu-64-bit/363879#363879

As seen in http://www.maketecheasier.com/run-32-bit-apps-in-64-bit-linux/

Used “Search the contents of packages” at http://packages.ubuntu.com/

Found in “Search the contents of packages” at http://packages.ubuntu.com/ using “lucid” distribution (instead of default “saucy”). This commit runs one `sudo apt-get -y install` for every package; otherwise each failure stops the whole installation process, but that makes the debugging painful (a test takes 13 minutes to run).

Two packages not found → removing.

The following `npm install` takes more than 3 seconds anyway; `xvfb` would already start before it becomes necessary for node-webkit.

Mithgol · 2014-02-13T19:11:53Z

**Achievement unlocked: ** 64bit and 32bit node-webkit can coexist on the same Travis CI Linux box within the same building&testing job.

Mithgol · 2014-02-13T19:24:37Z

@springmeyer You can land all of the above.

There's still some room for improvement (node-pre-gyp does not actually test node-sqlite3 on node-webkit because, unlike Node.js, node-webkit does not run javascripts such as ./lib/sqlite3 directly — and thus a dedicated application is necessary), but that improvement will happen elsewhere (in node-pre-gyp, to be precise).

springmeyer · 2014-02-13T19:39:28Z

Okay, excellent work, thank you! Would you mind if I merged manually? I'd like to take a shot at now at moving the script out of the travis.yml and into separate scripts. As far as the testing, I found that on OS X at least running ./lib/sqlite3 opened the code in a node-webkit browser window, which was odd, but seemed like a reasonable test. I can see how this would not work the same on other platforms since it is not really the correct usage :)

Mithgol · 2014-02-13T19:56:02Z

I'd recommend merging this automatically and then starting to build a less monolithic approach:

you'd have a point to revert to (if anything breaks),
the above trial-and-error process is recorded (with decisions and answers ready for questions such as “why so long, why not just install ia32-libs?” or “where in the web can we search the contents of packages?”) even if I eventually kill my fork.

Mithgol · 2014-02-13T20:01:11Z

Opening ./lib/sqlite3 as a plain text is not actually a test for anything. (Even on OS X.) It does not check if the file contains any valid JavaScript code, and I doubt it reacts differently even if a missing file is encountered.

There should be a real node-webkit-based application (as a part of node-pre-gyp) that would at least attempt to require() the module and exit (returning 1) if anything goes wrong.

springmeyer · 2014-02-13T21:00:11Z

okay, sounds good on merging as is and next actions: 1) breaking travis.yml apart into managable parts, and 2) actually testing node-webkit application rather than lib/sqlite3. As far as the latter I would prefer this not be the responsibility of node-pre-gyp, but rather done outside somehow. So basically node-pre-gyp testpackage would become a noop for node-webkit binaries and the developer would be responsible for testing via some other mechanism.

Build node-sqlite3 for node-webkit on Travis CI

Mithgol · 2014-02-21T07:06:40Z

@springmeyer

The big gap here is windows. Want to put thought into that. Really wish Travis supported windows, what alternatives do we have?

AppVeyor CI 2.0 beta testing was announced on February 19.

It says “Free plan with support of public repositories only”.

springmeyer · 2014-02-25T19:28:01Z

@Mithgol nice find. looks promising!

Mithgol · 2014-02-26T04:43:55Z

Created #261 to follow up the Windows issue on a separate page.

Mithgol added 3 commits February 6, 2014 02:52

Travis CI: non-empty NODE_WEBKIT="0.8.3" is set for Node v0.10

a9555b4

There's currently no API to determine the most recent version of node-webkit over the Web.

install nw-gyp in 32 bit environment if NODE_WEBKIT is not empty

31a11de

Note: `make clean` happens after `npm install nw-gyp -g` because we intend to use `nw-gyp` later to rebuild the existing build of node-sqlite3.

Travis CI: rebuild for the node-webkit (version $NODE_WEBKIT)

6903cde

Mithgol mentioned this pull request Feb 6, 2014

Provide node-sqlite3 builds for node-webkit #251

Closed

Mithgol added 8 commits February 6, 2014 04:24

build for node-webkit 64bit on Linux (uninstall nw-gyp later)

20ed666

attempt a build for node-webkit 0.8.4 (instead of 0.8.3)

c894e47

Release notes https://groups.google.com/d/msg/node-webkit/bPjur7aFaQc/4TP6o3SM-8sJ say that 0.8.4 fix improves the behaviour on OS X.

do not uninstall nw-gyp on 64 bit Linux

4f6db19

It seems (in installation logs) that nw-gyp does not contain any binary (i.e. compiled) addons. Thus it's fine if nw-gyp survives 64bit→32bit switch of our Node.js environment on Linux.

added --runtime=node-webkit --overwrite twice

e3324ce

Try a feature originally introduced in node-pre-gyp v0.4.2 commit mapbox/node-pre-gyp@370024a for mapbox/node-pre-gyp#14.

oops, the previous commit requires node-pre-gyp v0.4.2 or newer

ff08389

use the tarball version of node-pre-gyp v0.4.2

7dc529b

Working around a temporal inavailability of node-pre-gyp v0.4.2. See mapbox/node-pre-gyp#35 for details.

use https (instead of http) in the value of remote_uri

b580a1d

Otherwise the following check fails: https://github.com/springmeyer/node-pre-gyp/blob/3d4632a7333d420d6423299b143f5391b626e904/lib/util/versioning.js#L75-79

Mithgol mentioned this pull request Feb 13, 2014

platform-dependent name of the node-webkit's executable mapbox/node-pre-gyp#36

Merged

Mithgol added 2 commits February 13, 2014 03:19

update node-pre-gyp's source branch

6655d7c

Commit Mithgol/node-pre-gyp@eaea6e1 is now included in the node-pre-gyp's source used here.

on Linux 64 bit: download, unpack, use and erase node-webkit

a86e300

for testing node-webkit, launch a virtual display

6fc9166

xvfb recipe: http://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-GUI-(e.g.-a-Web-browser) sleep 3 recipe: http://docs.travis-ci.com/user/gui-and-headless-browsers/#Starting-a-Web-Server

on Linux 32 bit: download, unpack, use and erase node-webkit

65280f0

Mithgol added 4 commits February 13, 2014 07:10

also install libx11-6

65be174

try installing libx11-dev

ccf56f4

sudo apt-get update --fix-missing

05d7f8c

More magic.

save and restore PATH

fcdc43a

Mithgol mentioned this pull request Feb 13, 2014

missing libX11.so.6 nwjs/nw.js#1566

Closed

attempt ldd

e370b9c

This commit also erases `sudo apt-get update --fix-missing` introduced in one of the previous commits, because it didn't help.

Mithgol added 2 commits February 13, 2014 08:58

use the primary node-pre-gyp's repository

54f460b

on Linux 64 bit: also ldd nw

8b83d45

Mithgol added 6 commits February 13, 2014 09:30

install libx11-6:i386

6dbb9d9

Follow the following suggestion: http://askubuntu.com/questions/363878/how-to-install-32-bit-matlab-in-ubuntu-64-bit/363879#363879

install ia32-libs

0713dce

As seen in http://www.maketecheasier.com/run-32-bit-apps-in-64-bit-linux/

install 32 bit stuff necessary for node-webkit 32 bit

354766a

Used “Search the contents of packages” at http://packages.ubuntu.com/

do not install libpangocairo:i386 and libpango:i386

9c555c3

Two packages not found → removing.

remove sleep 3

e367dac

The following `npm install` takes more than 3 seconds anyway; `xvfb` would already start before it becomes necessary for node-webkit.

springmeyer pushed a commit that referenced this pull request Feb 13, 2014

Merge pull request #252 from Mithgol/support-node-webkit

c4dff8b

Build node-sqlite3 for node-webkit on Travis CI

springmeyer merged commit c4dff8b into TryGhost:master Feb 13, 2014

Mithgol mentioned this pull request Nov 12, 2016

Latest Version of nwjs supported #727

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build node-sqlite3 for node-webkit on Travis CI #252

Build node-sqlite3 for node-webkit on Travis CI #252

Mithgol commented Feb 6, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 21, 2014

springmeyer commented Feb 25, 2014

Mithgol commented Feb 26, 2014

Build node-sqlite3 for node-webkit on Travis CI #252

Build node-sqlite3 for node-webkit on Travis CI #252

Conversation

Mithgol commented Feb 6, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 13, 2014

Mithgol commented Feb 13, 2014

springmeyer commented Feb 13, 2014

Mithgol commented Feb 21, 2014

springmeyer commented Feb 25, 2014

Mithgol commented Feb 26, 2014