Use CI to build release binaries

[squeek502]

Currently only have a few precompiled binaries for Luvi's newer releases. We've been meaning to set this up for a while but never gotten around to it. Example of when release deploying was added to luv via travis: luvit/luv@6b901a2

Related issues: #187, #181

Here's what we had in v2.7.6: https://github.com/luvit/luvi/releases/tag/v2.7.6

• Windows Use appveyor to build/release Windows binaries #201
• Linux Use travis to build/release Linux binaries  #202
  • i686
  • x86_64
• Mac/Darwin active travis auto publish release #203
• ARM
  • aarch64 / arm64
  • armv7
  • armv6
• FreeBSD

[creationix]

Yes, this would be great if it could be automated. I think travis now has OSX guests and we have appveyor for windows, so it should be possible for most of these.

I was building freebsd and linux-686 in virtualbox images on my workstation, but I don't think those are as needed as others if it proves to be a problem.

[zhaozg]

Enough if win-x64,Mac-x64,Linux-x64 auto github-release for ci.Others are optional

[creationix]

Sure, that would be the bare minimum. Next important on the list I think would be arm32 and arm64 variants. Those platforms are getting more and more popular.

[zhaozg]

That means arm platform need manual release, I do not have news about arm free CI platform

[creationix]

Sometimes you can build them inside docker or chroots using qemu-user-static to emulate arm CPUs

[squeek502]

I'm going to start working on the Windows targets now. Will do the Linux/Mac ones after that.

[squeek502]

For Linux:

• Got something mostly working for x86_64 and i686 using travis, docker, and holy-build-box (master...squeek502:master / example release) but the resulting binaries are gigantic--need to look into why that is. EDIT: It's debug info.
  EDIT#2: Fixed and made a PR: Use travis to build/release Linux binaries  #202

For Mac:

• Right now we don't have Mac CI turned on at all, and I don't have a Mac to do any testing, so will probably need some help with this when I get around to it

For ARM:

• Funnily enough, the top google result for "travis arm" (for me) is an old article about getting Travis to cross-compile Luvit (1.0) for ARM: https://www.tomaz.me/2013/12/02/running-travis-ci-tests-on-arm.html
• It might be easier now to use Docker; more possible solutions here: Support Android / Arm travis-ci/travis-ci#3376

For FreeBSD:

• I know sourcehut.org/sr.ht offers FreeBSD CI and it should be possible to integrate that with Github in some way (seen it mentioned that it's possible before). EDIT: More info: https://man.sr.ht/tutorials/builds.sr.ht/github-integration.md

[zhaozg]

@squeek502 I'll continue to finish Mac base your code.

[zhaozg]

The PR #203 add auto publish for osx.

here is ci log. here is real effect

Notice: travis need GITHUB_OAUTH_TOKEN to do deploy.

[squeek502]

Nice, thanks @zhaozg. Should we release v2.9.3 to try out all the new automatic publishing?

EDIT: Probably makes sense to merge #204 before a new release

[squeek502]

Here goes nothing:

• https://github.com/luvit/luvi/releases/tag/v2.9.3
• https://travis-ci.org/luvit/luvi/builds/513312262
• https://ci.appveyor.com/project/racker-buildbot/luvi/builds/23468810

EDIT: Appveyor failed due to auth_token:

Error creating GitHub release: Provider setting not found or it's value is empty. If secure setting is used please check that value was encrypted (or YAML was exported) while being logged under correct account.
Parameter name: auth_token

Either I messed something up or we'll need to use an auth token from a different account than mine.

[zhaozg]

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

[squeek502]

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

I went with a different approach, and appveyor seems to be working now: https://github.com/luvit/luvi/releases/tag/v2.9.3-temp

Will clean things up once the build finishes: I'll move the Windows binaries to the v2.9.3 tag and delete the v2.9.3-temp tag. Then I'll revert c6a410e

[zhaozg]

Sorry for that, I will do nothing in next 4 hours. Please re tag v2.9.3 --force

[squeek502]

Ok, I think we're good to go. The Windows binaries have been moved to the v2.9.3 tag and still have the proper version (luvi.exe v2.9.3), and the v2.9.3-temp tag has been deleted. The v2.9.3 tag seems fine to me, please verify that it's ok. And any future releases should work correctly (I think what happened is that I accidentally had copied some whitespace with the auth_token I encrypted the first time).

Here's an example of the random build error I mentioned in #202 btw: https://travis-ci.org/luvit/luvi/jobs/513318947

[zhaozg]

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

• luvi-regular-Darwin_x86_64
• luvi-regular-Linux_i686
• luvi-regular-Linux_x86_64
• luvi-regular-Windows-amd64.exe
• luvi-regular-Windows-amd64.lib
• luvi-regular-Windows-ia32.exe
• luvi-regular-Windows-ia32.lib
• luvi-tiny-Darwin_x86_64
• luvi-tiny-Linux_i686
• luvi-tiny-Linux_x86_64
• luvi-tiny-Windows-amd64.exe
• luvi-tiny-Windows-ia32.exe

[squeek502]

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

Tested the rest except the Windows .lib files. Everything seems good.

[zhaozg]

to @squeek502
I think change https://github.com/luvit/luvi/blob/master/.travis.yml#L47 to https://github.com/luvit/luvi/blob/master/.travis.yml#L69, and remove https://github.com/luvit/luvi/blob/master/.travis.yml#L48-L49 will also do publish with out docker,

[squeek502]

Using docker & holy-build-box has portability benefits: https://github.com/phusion/holy-build-box

[creationix]

I've had good experience using docker combined with qemu-user-static to build stuff in arm containers on x86 kernels/hosts

[squeek502]

Looks like we can use Github actions to build arm binaries. @truemedian is doing just that here:

https://github.com/truemedian/luvit-bin/blob/main/.github/workflows/manual.yml#L154-L196

[truemedian]

My approach in luvit-bin for ARM binaries is.. not the best, nor would I recommend it. It requires running a full system vm on github's actions machines, which turns a 2-3 minute build process into a 30-40 minute build process

[squeek502]

Surely it's better than nothing. If we don't want to take the hit for every CI, we could only build arm for tagged releases.

[truemedian]

I decided to take another stab at this for building just luvi (this can be rather easily extended to both lit and luvit if need be).

ARM + OpenSSL binaries take the longest, so thats what these times are going to based off of.

from scratch: 37m 31s.
full ccache: 9m 56s.

Surprisingly, this is about on par with the current CI setup in terms of speed (when the ccache is full, having to compile slows down the process massively for ARM builds)

This allows for multiple things:

• All (or most of) the CI in one place.
• Running tests on PRs to prevent serious breakages.
• +armv6/armv7 builds with "minimal effort"

A few things of note:

• arm64v8 (aarch64) is currently unbuildable due to some issue in arm64 luajit (anyone want to pin it down? its a segfault / "attempt to compare two table values", possibly related)
• no FreeBSD support