CI options

[ppenzin]

Given our troubles with CI on Microsoft/ChakraCore, we can try to set up CI on the clone of CC code we have in this org. This would need to be done at some point anyway to migrate the repo here.

There are is a number of cloud CI solutions, though I only have experience real experience with Travis and some experience interacting with AppVeyor and Azure.

Let me know what you think.

Azure DevOps

That is what CC uses currently, so it can be an obvious choice to just continue with it. Adwantages are lax limits on build time and memory use and Windows support. The big disadvantage is that it seems go be buggy, and requires manual restarts. If we have access we can restart it though, plus we don't have to test on Windows 7 and 8.

Travis CI

In my experience it is pretty realiable, but it used to have memory and time limits for free tier, which we might not fit into. Also, not sure about Windows support.

Non-cloud CI

ChakraCore is a relatively large project (not as large as full browser would be, but still), and for larger projects it is often cheaper to have physical build machines, with only CI master being in the cloud. For example LLVM uses Buildbot, with actual build servers across a few companies attached to it. We would not require as much of an investment, of course. We can set up a Buildbot instance somewhere (I've seen VM services where base tier is $5 per month) and hook up local machines to it.

One question would be procuring the build machines - I can maybe find parts for a Linux or Windows box (or both), but Mac would be harder. However, advantage of this approach is that in the future people can "volunteer" their machines to be Buildbot's workers.

The other open question is integrating with Github - it seems to be possible, but I haven't done it. I do have a set up like this for my Fortran compiler though.

[Fly-Style]

I had experience with Travis CI. I think it shouldn't be so hard to setup.

[rhuanjl]

I've never set up a CI system, so I'm probably the least qualified to comment. That said, I have fairly detailed knowledge of how to run all the tests from the test suite locally on windows, linux and macOS so can explain that part if needed to assist anyone setting up new CI.

Whatever we have we'd want:

1. Testing for Linux, macOs and Windows
2. Testing of a couple of different flavours etc. (similar to existing)

Obviously if anything costs money that opens a question of the fact that we have 0 financial budget at the moment - I could cover something like $5 but if it's going to go up from that then maybe not

As a related point
We want to get fuzzing going again when we can, potentially with OSS-Fuzz, not something I know much about except that that they used to fuzz chakracore and only stopped as it was unmaintained, see: google/oss-fuzz#3545 and google/oss-fuzz#3491

[rhuanjl]

Had a quick look at Travis and Azure.

Travis gives you 5 free concurrent jobs for open source projects but has bad windows support. Azure has 10 free concurrent jobs for open source.

At least on paper Azure looks nicer - also I wonder if a hybrid approach with github actions doing style checks and Azure for the builds may be better to pull some of the burden off of Azure. Obviously we've had pain with Azure when we haven't had access to it but if we had full access it probably wouldn't be as bad?

[ppenzin]

If we have access to reboot the CI, then Azure is not as bad as the set up we have now. I think Travis is probably more reliable than Azure, though I haven't looked into its ability to run Windows builds. It might make sense to try both and see which finishes full CI for a PR faster :)

What do you think about supporting specific Windows versions, like Win 7 or Win 8?