Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: Transition from TravisCI to Github Actions #329

Closed

Conversation

drakenclimber
Copy link
Member

@drakenclimber drakenclimber commented Jul 21, 2021

This pull request resolves Issue #299 and Issue #312.

This patchset transitions the libseccomp continuous integration from TravisCI to Github Actions. Here's a brief rundown of the pros/cons of each CI system:

TravisCI Github Actions
Run Time1 ~24 min ~17 min
Architectures Being Tested2 amd64, arm64, ppc64le amd64
Parallelization Each architecture is a separate run Each test type (simulation, live, scan build, code coverage) is a separate run
Reliability Excellent Excellent
When Does it Run On commits and pull requests On commits and pull requests
Costs Uncertain. Travis is still claiming free support for open source, but it looks both tedious and tenuous Free. We've had no issues in libcgroup so far
User Experience - debugging failures Must wade through the logs. Likely tedious for occasional libseccomp contributors Each test type is split into a separate run, and the GUI clearly delineates them. A live test failure vs. a simulation failure is obvious, but a user still needs to dig into logs for the exact cause
Code Coverage3 Our current TravisCI code coverage uses a user-generated coveralls wrapper that sends the coverage data to coveralls.io. It's worked well, but is a little nonstandard The proposed Github Actions solution uses a Github Action developed by Coveralls, and in theory has better support and integration with Github Actions and Coveralls.
Current Code Coverage Numbers 93.0% 89.9%

1 It seems that the TravisCI VMs are more powerful than the Github Actions VMs, but Github Actions makes parallelization much easier. Thus the nontrivial run time differences.
2 There are a few ways to add support for additional architectures to Github Actions via either self-hosted runners or possibly this Docker container solution
3 The code coverage numbers differ slightly from the TravisCI implementation to the Github Actions implementation. I think the Github Actions numbers are more accurate, but I haven't dug too deeply. (Also, I can't get Github Actions code coverage to ignore arch-syscall-check.c. It's kinda bugging me :).)

Disable Travis CI by deleting the .travis.yml file.  Subsequent
commits will enable the Github Actions continuous integration.

Signed-off-by: Tom Hromatka <tom.hromatka@oracle.com>
Delete the unused variable 'len' from scmp_bpf_disasm.

scan-build identified the following two warnings:

    scmp_bpf_disasm.c:304:10: warning: Although the value stored to 'len'
    is used in the enclosing expression, the value is never actually read
    from 'len'
            while ((len = fread(&bpf, sizeof(bpf), 1, file))) {

    scmp_bpf_disasm.c:441:10: warning: Although the value stored to 'len' is
    used in the enclosing expression, the value is never actually read from
    'len'
            while ((len = fread(&bpf, sizeof(bpf), 1, file))) {

Signed-off-by: Tom Hromatka <tom.hromatka@oracle.com>
@drakenclimber drakenclimber changed the title Transition from TravisCI to Github Actions RFE: Transition from TravisCI to Github Actions Jul 21, 2021
@drakenclimber drakenclimber added this to the v2.5.2 milestone Jul 21, 2021
@pcmoore
Copy link
Member

pcmoore commented Jul 21, 2021

I haven't looked at the changes/commits yet, but +1000 on the ultra snazzy PR description!

.github/actions/setup/action.yml Outdated Show resolved Hide resolved
.github/actions/setup/action.yml Outdated Show resolved Hide resolved
@pcmoore
Copy link
Member

pcmoore commented Jul 21, 2021

@drakenclimber out of curiosity, what is the code coverage percentage with the GH Actions approach?

@drakenclimber
Copy link
Member Author

@drakenclimber out of curiosity, what is the code coverage percentage with the GH Actions approach?

I added a link to each in the table above.
93.0% - TravisCI
89.9% - Github Actions

Add Github Actions workflow and actions to run the automated
libseccomp tests and gather code coverage metrics.

Signed-off-by: Tom Hromatka <tom.hromatka@oracle.com>
Travis CI has now been disabled.  Delete the Travis CI
build status badge and display the Github Actions
continuous integration workflow badge.

Signed-off-by: Tom Hromatka <tom.hromatka@oracle.com>
@pcmoore
Copy link
Member

pcmoore commented Jul 27, 2021

@drakenclimber out of curiosity, what is the code coverage percentage with the GH Actions approach?

I added a link to each in the table above.
93.0% - TravisCI
89.9% - Github Actions

Arrrgh less than 90%?! Several years ago I spent an absurd amount of time one year getting the coverage above 90%, seeing it drop below 90% is a sad moment :) Although most of that was due to fixing various small bugs that cropped up during the additional testing so I can't be too upset.

Although looking at the coverage data there are definitely a few things that look .... off. For example, the file list is definitely not correct (files missing, others added that shouldn't be there, etc.). Regardless, that's all something we can fix later.

@pcmoore
Copy link
Member

pcmoore commented Jul 27, 2021

Merged at HEAD 04245d9.

I'm thinking that maybe we keep this issue open until we sort out some of the remaining code coverage issues and backport everything to the release-2.5 branch, or should we close this out and create a new issue for that - thoughts?

@pcmoore
Copy link
Member

pcmoore commented Aug 12, 2021

With this merged into the main branch and PR #331 backporting this to the release-2.5 branch I think we can close this out now, any remaining issues should be treated as new issues or handled in the backport PR.

@pcmoore pcmoore closed this Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RFE: Transition from travis-ci.org to travis-ci.com BUG: look into replacing Travis CI with GitHub actions
2 participants