Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: refactor tests to separate out direct NVD connections #4044

Open
terriko opened this issue Apr 17, 2024 · 2 comments
Open

CI: refactor tests to separate out direct NVD connections #4044

terriko opened this issue Apr 17, 2024 · 2 comments
Assignees
Labels
CI Related to our continuous integration service (GitHub Actions)
Milestone

Comments

@terriko
Copy link
Contributor

terriko commented Apr 17, 2024

I'm collapsing a few older issues into a single one:

Our tests currently use a cache of NVD that's stored in github, and we've done some work in separating out places where we need network connections, but it's been some time since we looked through it all. We aren't currently using our own mirror for much because it was developed after the tests were last revamped, but if we turn off the NVD_API_KEY on some jobs it will default to that.

I'd like to think about how we should refactor the tests to

  • reduce codecov jitter (Improve usage of NVD_API_KEY and Codecov #1961 )
  • reduce the number of tests that explicitly need an NVD_API_KEY set (We may want them all in the network-mayfail job, or we may need a windows equivalent of this job)
  • Divide out the jobs that can't be run on the cache (Refactoring NVD update tests to avoid network connections. #1977) -- these should be all in the network-mayfail job already but I don't think they are.
  • figure out if we should be using the mirror directly in any tests. Maybe the network ones as much as possible?
  • improve the stability of the windows test job if possible, which currently has more network fails than the linux jobs (seems to be some sort of issue with the github actions framework for windows). This could involve splitting it into some pieces so the network fails wouldn't affect the rest of hte job, or the long parts of the job (mostly the installs from conda) could be moved into something that could run in parallel.

This is an issue best tackled by someone who's gotten some experience using the test suite, so probably not suitable for a beginner but anyone who's got a few PRs under their belt likely has an opinion about the test suite and could take a stab at it.

@terriko terriko added the CI Related to our continuous integration service (GitHub Actions) label Apr 17, 2024
@terriko
Copy link
Contributor Author

terriko commented Apr 17, 2024

While refactoring, we might also want to consider what we should do about potentially separating our test db and our scanning db going forwards:

@terriko terriko added this to the 3.3.1 milestone Apr 17, 2024
@mastersans
Copy link
Contributor

@terriko I'll like to work on this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Related to our continuous integration service (GitHub Actions)
Projects
None yet
Development

No branches or pull requests

2 participants