fix: Timeout Error with CVE-Bin-Tool Database Update/Download

[tvyavaha]

Description

I'm experiencing a timeout error when updating/downloading the database using cve-bin-tool --nvd api2, even with the nvd_key option and --nvd json-mirror. This issue persists across different networks and has prevented me from using the tool effectively.

Also I tried --disable-data-source EPSS option, but still getting the same timed out error.

To reproduce

Steps to reproduce the behaviour:

1. Run this command: cve-bin-tool -l debug --nvd json-mirror -u now --disable-data-source EPSS
   Expected behaviour: No time out error.
   Actual behaviour:

x: ~>cve-bin-tool --nvd json-mirror -u now --disable-data-source EPSS
[06:57:24] INFO     cve_bin_tool - CVE Binary Tool v3.3.1dev0                                                                                                                                                                                                                                                   cli.py:571
           INFO     cve_bin_tool - This product uses the NVD API but is not endorsed or certified by the NVD.                                                                                                                                                                                                   cli.py:572
           INFO     cve_bin_tool - Disabling data source EPSS                                                                                                                                                                                                                                                   cli.py:693
[06:57:55] WARNING  cve_bin_tool.CVEDB - Updating cachedir /home/tvyavaha/.cache/cve-bin-tool                                                                                                                                                                                                                 cvedb.py:665
[06:58:03] INFO     cve_bin_tool - You are running version 3.3.1dev0 of cve-bin-tool but the latest PyPI Version is 3.3.                                                                                                                                                                                     version.py:27
           INFO     cve_bin_tool - Getting NVD CVE data...                                                                                                                                                                                                                                               nvd_source.py:327
           INFO     cve_bin_tool - Getting GitLab Advisory Database CVEs...                                                                                                                                                                                                                               gad_source.py:86
           INFO     cve_bin_tool.CVEDB - Rolling back the cache to its previous state                                                                                                                                                                                                                         cvedb.py:809
           INFO     cve_bin_tool - Getting RedHat CVEs...                                                                                                                                                                                                                                              redhat_source.py:69
Downloading CVEs... ________________________________________ 100% 0:00:00
[07:01:26] INFO     cve_bin_tool - Getting Open Source Vulnerability Database CVEs...                                                                                                                                                                                                                    osv_source.py:161
[07:01:51] ERROR    cve_bin_tool - Unable to fetch OSV CVEs, skipping OSV.                                                                                                                                                                                                                               osv_source.py:376
________________________________ Traceback (most recent call last) _________________________________
_ /usr/local/bin/cve-bin-tool:8 in <module>                                                        _
_                                                                                                  _
_   5 from cve_bin_tool.cli import main                                                            _
_   6 if __name__ == '__main__':                                                                   _
_   7 _   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         _
_ _ 8 _   sys.exit(main())                                                                         _
_   9                                                                                              _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/cli.py:808 in main                          _
_                                                                                                  _
_    805 _                                                                                         _
_    806 _   # update db if needed                                                                 _
_    807 _   if db_update != "never":                                                              _
_ _  808 _   _   cvedb_orig.get_cvelist_if_stale()                                                 _
_    809 _   else:                                                                                 _
_    810 _   _   LOGGER.warning("Not verifying CVE DB cache")                                      _
_    811 _   _   if not cvedb_orig.check_cve_entries():                                            _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/cvedb.py:281 in get_cvelist_if_stale        _
_                                                                                                  _
_    278 _   _   _   datetime.datetime.today()                                                     _
_    279 _   _   _   - datetime.datetime.fromtimestamp(self.dbpath.stat().st_mtime)                _
_    280 _   _   ) > datetime.timedelta(hours=24):                                                 _
_ _  281 _   _   _   self.refresh_cache_and_update_db()                                            _
_    282 _   _   _   self.time_of_last_update = datetime.datetime.today()                          _
_    283 _   _   else:                                                                             _
_    284 _   _   _   _ = self.get_db_update_date()                                                 _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/cvedb.py:268 in refresh_cache_and_update_db _
_                                                                                                  _
_    265 _   _                                                                                     _
_    266 _   _   # if the database isn't open, open it                                             _
_    267 _   _   self.init_database()                                                              _
_ _  268 _   _   self.populate_db()                                                                _
_    269 _   _   self.LOGGER.debug("Updating exploits data.")                                      _
_    270 _   _   self.create_exploit_db()                                                          _
_    271 _   _   self.update_exploits()                                                            _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/cvedb.py:483 in populate_db                 _
_                                                                                                  _
_    480 _   _                                                                                     _
_    481 _   _   # EPSS uses metrics table to get the EPSS metric id.                              _
_    482 _   _   # It can't be run before creation of metrics table.                               _
_ _  483 _   _   self.populate_epss()                                                              _
_    484 _   _   self.store_epss_data()                                                            _
_    485 _   _                                                                                     _
_    486 _   _   for idx, data in enumerate(self.data):                                            _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/cvedb.py:635 in populate_epss               _
_    632 _   _   Add EPSS data into the database"""                                                _
_    633 _   _   epss = epss_source.Epss_Source()                                                  _
_    634 _   _   cursor = self.db_open_and_get_cursor()                                            _
_ _  635 _   _   self.epss_data = run_coroutine(epss.update_epss(cursor))                          _
_    636 _   _   self.db_close()                                                                   _
_    637 _                                                                                         _
_    638 _   def metric_finder(self, cursor, cve):                                                 _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/async_utils.py:90 in run_coroutine          _
_                                                                                                  _
_    87 _   """                                                                                    _
_    88 _   loop = get_event_loop()                                                                _
_    89 _   aws = asyncio.ensure_future(coro, loop=loop)                                           _
_ _  90 _   result = loop.run_until_complete(aws)                                                  _
_    91 _   return result                                                                          _
_    92                                                                                            _
_    93                                                                                            _
_                                                                                                  _
_ /usr/lib/python3.11/asyncio/base_events.py:653 in run_until_complete                             _
_                                                                                                  _
_    650 _   _   if not future.done():                                                             _
_    651 _   _   _   raise RuntimeError('Event loop stopped before Future completed.')             _
_    652 _   _                                                                                     _
_ _  653 _   _   return future.result()                                                            _
_    654 _                                                                                         _
_    655 _   def stop(self):                                                                       _
_    656 _   _   """Stop running the event loop.                                                   _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/data_sources/epss_source.py:47 in           _
_ update_epss                                                                                      _
_                                                                                                  _
_    44 _   _   _   _     - EPSS percentile                                                        _
_    45 _   _   """                                                                                _
_    46 _   _   self.EPSS_id_finder(cursor)                                                        _
_ _  47 _   _   await self.download_and_parse_epss()                                               _
_    48 _   _   return self.epss_data                                                              _
_    49 _                                                                                          _
_    50 _   async def download_and_parse_epss(self):                                               _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/data_sources/epss_source.py:52 in           _
_ download_and_parse_epss                                                                          _
_                                                                                                  _
_    49 _                                                                                          _
_    50 _   async def download_and_parse_epss(self):                                               _
_    51 _   _   """Downloads and parses the EPSS data from the CSV file."""                        _
_ _  52 _   _   await self.download_epss_data()                                                    _
_    53 _   _   self.epss_data = self.parse_epss_data()                                            _
_    54 _                                                                                          _
_    55 _   async def download_epss_data(self):                                                    _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/cve_bin_tool/data_sources/epss_source.py:93 in           _
_ download_epss_data                                                                               _
_                                                                                                  _
_    90 _   _   else:                                                                              _
_    91 _   _   _   try:                                                                           _
_    92 _   _   _   _   async with aiohttp.ClientSession(headers=HTTP_HEADERS) as session:         _
_ _  93 _   _   _   _   _   async with session.get(self.DATA_SOURCE_LINK) as response:             _
_    94 _   _   _   _   _   _   response.raise_for_status()                                        _
_    95 _   _   _   _   _   _   self.LOGGER.info("Getting EPSS data...")                           _
_    96 _   _   _   _   _   _   decompressed_data = gzip.decompress(await response.read())         _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/aiohttp/client.py:1197 in __aenter__                     _
_                                                                                                  _
_   1194 _   _   return self.__await__()                                                           _
_   1195 _                                                                                         _
_   1196 _   async def __aenter__(self) -> _RetType:                                               _
_ _ 1197 _   _   self._resp = await self._coro                                                     _
_   1198 _   _   return self._resp                                                                 _
_   1199                                                                                           _
_   1200                                                                                           _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/aiohttp/client.py:507 in _request                        _
_                                                                                                  _
_    504 _   _                                                                                     _
_    505 _   _   timer = tm.timer()                                                                _
_    506 _   _   try:                                                                              _
_ _  507 _   _   _   with timer:                                                                   _
_    508 _   _   _   _   while True:                                                               _
_    509 _   _   _   _   _   url, auth_from_url = strip_auth_from_url(url)                         _
_    510 _   _   _   _   _   if auth and auth_from_url:                                            _
_                                                                                                  _
_ /usr/local/lib/python3.11/dist-packages/aiohttp/helpers.py:735 in __exit__                       _
_                                                                                                  _
_    732 _   _   _   self._tasks.pop()                                                             _
_    733 _   _                                                                                     _
_    734 _   _   if exc_type is asyncio.CancelledError and self._cancelled:                        _
_ _  735 _   _   _   raise asyncio.TimeoutError from None                                          _
_    736 _   _   return None                                                                       _
_    737 _                                                                                         _
_    738 _   def timeout(self) -> None:                                                            _
____________________________________________________________________________________________________
TimeoutError
x: ~>

Version/platform info

Version of CVE-bin-tool( e.g. output of cve-bin-tool --version): 3.3.1dev0
Installed from pypi or github? pypi
Operating system: Linux/Windows (other platforms are unsupported but feel free to report issues anyhow)

• On Linux (or Windows Subsystem for Linux) you can run uname -a
  x: ~>uname -a
  Linux x 6.9.0-rc1+ SMP PREEMPT_DYNAMIC Fri Mar 29 08:33:41 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Python version (e.g. python3 --version): Python 3.11.6
Running in any particular CI environment we should know about? (e.g. Github Actions) No.

[terriko]

Darn, I was hoping that workaround will work.

Fixing this should just involve fixing it so we don't get a stack trace if there's a timeout error, but we might want to also double-check that there's nothing weird with the link we're using for epss data.

[terriko]

I've tried a couple of things but I can't seem to duplicate this myself easily even with an empty database. As tempting as it is to start messing with my routing table to see if I can make it happen, I think it probably makes more sense to just catch the timeout and hope we can work around it with me going in a bit blind? So I'm going to try to do that instead of bashing my head against it trying to figure out why it's happening.

[terriko]

So it turns out the reason --disable-data-source EPSS didn't work is that EPSS wasn't loading in the same way as the other data sources. Le sigh. Fixing it so it could be disabled wasn't too hard, but fixing it so it behaves exactly like the other data sources (including better handling of the timeouts) will take a bit more refactoring. I'm testing a halfway between solution right now to see if it's worth having a halfway solution or if I should just refactor properly.

[terriko]

I've got a tentative fix in #4125 that may work for you. It should at least allow the EPSS data source to be disabled correctly as a workaround, as well as allow it to fail more gracefully. I haven't run it against the full test suite yet so it may need a bit of work still.