Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add printouts for network calls #746

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

kemzeb
Copy link
Contributor

@kemzeb kemzeb commented Jan 15, 2024

Since in #698 it was deemed that a dry run mode may not be feasible, it was determined that printing these network calls during runtime maybe a good alternative.

I'm not quite sure if this is still desired, but I believe this would add more transparency as to what the runtime of OSV-Scanner is doing without the need to look at the code or docs. While implementing this, the following guided my thought process:

  1. What endpoints should be printed?
  2. Where in the codebase should we add these printouts?
  3. How should we printout these network calls?
  4. How can we avoid creating expensive messages when users don't wish to print messages at the VerboseLevel?

For point 3, I wasn't too sure how to display the data, so I decided to marshall it into JSON and took a page out of curl's book and prefixed these messages with > and < to represent an HTTP request and response respectfully. I also prefixed the JSON result for better readability. As a side note I have been using the osv.dev API docs as a reference.

For point 4, I went with doing lazy evaluation. I implemented a CanPrintLevel() method in the Reporter interface and used it to determine if I should create these expensive messages. I also used this method across the Warnf, Errorf, and other methods to try and keep things DRY. Another approach I thought of was to have a PrintFunc(lvl VerbosityLevel, func() string) where callers would need to build these expensive messages in the callback (and we check lvl to see if we even should build it), but the CanPrintLevel() approach seems simpler.

If there is interest, I'll go ahead and try to tackle the following endpoints as well:

  • The determine version HTTP GET endpoint
  • The license checking gRPC endpoint
  • The local database-related HTTP endpoints (i.e. the HEAD and GET endpoints)

@codecov-commenter
Copy link

codecov-commenter commented Jan 15, 2024

Codecov Report

Attention: 46 lines in your changes are missing coverage. Please review.

Comparison is base (c75d056) 79.08% compared to head (6ad567e) 78.54%.
Report is 2 commits behind head on main.

Files Patch % Lines
pkg/osvscanner/osvscanner.go 12.00% 42 Missing and 2 partials ⚠️
pkg/reporter/void_reporter.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #746      +/-   ##
==========================================
- Coverage   79.08%   78.54%   -0.55%     
==========================================
  Files          86       86              
  Lines        6121     6181      +60     
==========================================
+ Hits         4841     4855      +14     
- Misses       1075     1119      +44     
- Partials      205      207       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kemzeb
Copy link
Contributor Author

kemzeb commented Jan 15, 2024

Here's a truncated example of how it currently prints out:

Scanning dir .
Scanning /workspaces/osv-scanner/ at commit 7310bc02536dd6f5adf92abb364bc42427e306ea
Scanned /workspaces/osv-scanner/docs/Gemfile.lock file and found 94 packages
Scanned /workspaces/osv-scanner/go.mod file and found 52 packages
Scanned /workspaces/osv-scanner/internal/sourceanalysis/fixtures-rust/rust-project/Cargo.lock file and found 1 package
Scanned /workspaces/osv-scanner/internal/sourceanalysis/integration/fixtures-go/test-project/go.mod file and found 4 packages
Scanning directory for vendored libs: /workspaces/osv-scanner/internal/thirdparty
Scanning potential vendored dir: /workspaces/osv-scanner/internal/thirdparty/ar
> POST https://api.osv.dev/v1/querybatch
> {
>   "queries": [
>     {
>       "commit": "7310bc02536dd6f5adf92abb364bc42427e306ea",
>       "package": {}
>     },
>     {
>       "package": {
>         "name": "activesupport",
>         "ecosystem": "RubyGems"
>       },
>       "version": "7.1.2"
>     },
. . .
>   ]
> }
< POST https://api.osv.dev/v1/querybatch
< {
<   "results": [
<     {
<       "vulns": null
<     },
<     {
<       "vulns": null
<     },
<     {
<       "vulns": null
<     },
. . .
<       "vulns": [
<         {
<           "id": "GO-2023-2375"
<         },
<         {
<           "id": "GO-2023-2041"
<         },
<         {
<           "id": "GO-2023-2043"
<         },
<         {
<           "id": "GO-2023-2102"
<         },
<         {
<           "id": "GO-2023-2185"
<         },
<         {
<           "id": "GO-2023-2186"
<         },
<         {
<           "id": "GO-2023-2382"
<         }
<       ]
<     }
<   ]
< }
> GET https://api.osv.dev/v1/vulns/GHSA-c3h9-896r-86jm
< GET https://api.osv.dev/v1/vulns/GHSA-c3h9-896r-86jm
< {
<   "modified": "2023-12-06T01:01:11Z",
<   "published": "2022-03-28T20:28:00Z",
<   "schema_version": "1.6.0",
<   "id": "GHSA-c3h9-896r-86jm",
<   "aliases": [
<     "BIT-consul-2021-3121",
<     "BIT-protobuf-2021-3121",
<     "CVE-2021-3121",
<     "GO-2021-0053"
<   ],
<   "summary": "Improper Input Validation in GoGo Protobuf",
. . .

@kemzeb kemzeb changed the title Implement printouts for network calls Add printouts for network calls Jan 18, 2024
@another-rex another-rex self-requested a review January 21, 2024 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants