Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nvdlib.searchCVE occasionally times out #26

Closed
johnlabuyfoy1024 opened this issue Jul 3, 2023 · 21 comments
Closed

nvdlib.searchCVE occasionally times out #26

johnlabuyfoy1024 opened this issue Jul 3, 2023 · 21 comments
Assignees
Labels
wontfix This will not be worked on

Comments

@johnlabuyfoy1024
Copy link

I'm using nvdlib-0.7.4 and occasionally I get a time out error from urllib3 when executing:
r = nvdlib.searchCVE(cveId=s2,key='xyz',verbose=True,delay=6)
Timeout is currently set to 30
Is there a way I can increase the timeout?
Thanks

@vehemont
Copy link
Owner

vehemont commented Jul 5, 2023

Hi,

Can you please provide the entire error output when the timeout occurs?

Thanks

Edit:
#26 (comment)
NVD API is having issues at the moment.

@vehemont vehemont self-assigned this Jul 5, 2023
@SSarka69
Copy link

SSarka69 commented Jul 6, 2023

Hi,

I am also facing similar timeout issue occasionally. Please find the attached picture for reference and is there any solution to this error?

issue_nvdlib

Thanks

@johnlabuyfoy1024
Copy link
Author

Code:

#Get the details of the CVE from NVD Database using nvdlib
r = [] #Initialize r
r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True)

Error Message:

Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22954
Traceback (most recent call last):
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/johnf/anaconda3/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1274, in recv_into
return self.read(nbytes, buffer)
File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1130, in read
return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 451, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/johnf/cve_manager_VS/./first-api-epss.py", line 169, in
r = nvdlib.searchCVE(cveId=s2,key='a3340ee7-6c27-4f56-a7b6-55cbafa0cf1e',verbose=True)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/cve.py", line 156, in searchCVE
raw = __get('cve', headers, parameters, limit, verbose, delay)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/get.py", line 23, in __get
raw = requests.get(link, params=stringParams, headers=headers, timeout=30)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 578, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

@vehemont
Copy link
Owner

vehemont commented Jul 7, 2023

Thank you for the output. Could you please see if you still experience the issue after adding delay=12 to nvdlib.searchCVE?

Example:
r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, delay=12)

This will add a 12 second delay to the search. NVD has firewall rules in place that could be dropping the connection (or putting up a 404) because it believes your IP address to be going over the rate limit. NVDLib has built-in rate limiting, but it can be bypassed unintentionally by stopping and starting code repeatedly that makes searches.

@johnlabuyfoy1024
Copy link
Author

Still getting timeout error -
Code:

#Get the details of the CVE from NVD Database using nvdlib
r = [] #Initialize r
r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, delay=12)

/Error message:

Now running first-api-epss.py
url = https://api.first.org/data/v1/epss?order=!epss&limit=1500
Wrote intermediate output-epss.txt
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-30525
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22947
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22965
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22954
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-26134
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-1388
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-22963
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-37061
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-35405
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-29464
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-26352
Filter:
https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2022-37042
Traceback (most recent call last):
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/home/johnf/anaconda3/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/johnf/anaconda3/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1274, in recv_into
return self.read(nbytes, buffer)
File "/home/johnf/anaconda3/lib/python3.10/ssl.py", line 1130, in read
return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 451, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/johnf/cve_manager_VS/./first-api-epss.py", line 169, in
r = nvdlib.searchCVE(cveId=s2,key='a3340ee7-6c27-4f56-a7b6-55cbafa0cf1e',verbose=True, delay=12)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/cve.py", line 156, in searchCVE
raw = __get('cve', headers, parameters, limit, verbose, delay)
File "/home/johnf/anaconda3/lib/python3.10/site-packages/nvdlib/get.py", line 23, in __get
raw = requests.get(link, params=stringParams, headers=headers, timeout=30)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/johnf/.local/lib/python3.10/site-packages/requests/adapters.py", line 578, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

@jacobocasado
Copy link

I get the same error. Could be a srv issue?

@vehemont
Copy link
Owner

vehemont commented Jul 10, 2023

nvdlib-0.7.5-py3-none-any.zip

Attached is a compressed wheel file that contains an update to allow a custom timeout parameter. By default it set to 30 seconds, but you can enter a custom value. I tested it when setting the value to 60 and it seems to work for me, but please try it and see if it resolves your issue. If it works, then I will release this version.

Example:
r = nvdlib.searchCVE(cveId=s2,key='xyzxyzxyzxyz',verbose=True, timeout=60)

To install the wheel, you must first right click and extract the contents, then navigate to the same directory as the wheel file and run the following. You might need to first uninstall the current nvdlib by running pip uninstall nvdlib then running the below:
python install nvdlib-0.7.5-py3-none-any.whl

I believe this is caused by the NVD server taking too long to reply, which isn't an issue as long as we an define an appropriate timeout.

@jacobocasado
Copy link

I'll try it today, thx!

@johnlabuyfoy1024
Copy link
Author

I tried nvdlib 0.7.5 and with timeout = 60 and some other settings as well and still get read timeout errors and HTTP 503 errors. However, I emailed the nvd today and got this response which seems to confirm that the problem is on the nvd side:

"Thank you for notifying the NVD. We are aware of sporadic errors being provided to users of the APIs and are investigating the root cause for resolution. We apologize for the inconvenience during this time."

@vehemont
Copy link
Owner

Thanks for reaching out to them. I will keep this issue open in the mean time so people can find updates.

@vehemont vehemont added the wontfix This will not be worked on label Jul 11, 2023
@vehemont vehemont pinned this issue Jul 11, 2023
@mhdawson
Copy link

We've been getting persistent timeouts like this

Run (
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/ssl.py", line 1242, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/ssl.py", line 1100, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 538, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='services.nvd.nist.gov', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 261, in <module>
    exit(main())
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line 236, in main
    nvd_vulnerabilities: list[Vulnerability] = query_nvd(
  File "/home/runner/work/nodejs-dependency-vuln-assessments/nodejs-dependency-vuln-assessments/dep_checker/main.py", line [15](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:16)5, in query_nvd
    for cve in searchCVE(
  File "/opt/hostedtoolcache/Python/3.9.[17](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:18)/x64/lib/python3.9/site-packages/nvdlib/cve.py", line 270, in searchCVE
    raw = __get('cve', headers, parameters, limit, verbose, delay)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/nvdlib/get.py", line [22](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:23), in __get
    raw = requests.get(link, params=stringParams, headers=headers, timeout=[30](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:31))
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/requests/adapters.py", line 5[32](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:33), in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='services.nvd.nist.gov', port=4[43](https://github.com/nodejs/nodejs-dependency-vuln-assessments/actions/runs/5491594759/jobs/10008272004#step:6:44)): Read timed out. (read timeout=30)
Error: Process completed with exit code 1.

Since Wednesday July 5th.

I upgraded to 0.7.4 today but that did not change the behaviour.

I then found this discussion and see the issue is likely on the nvd side. Do those on the thread think it is worth adjusting the timeout (which is 6) etc. or just waiting until we hear the the problem on the nvd side is resolved.

@vehemont
Copy link
Owner

vehemont commented Jul 13, 2023

Do those on the thread think it is worth adjusting the timeout (which is 6) etc. or just waiting until we hear the the problem on the nvd side is resolved.

From the evidence in this thread, changing the timeout does not make any difference. It will be best to wait for an update from NVD.

@mhdawson
Copy link

@vehemont thanks for confirming.

@Fares-Harri
Copy link

Fares-Harri commented Jul 17, 2023

using nvdlib 0.7.4, for around 10 days it is not working for me and now I tested this
https://services.nvd.nist.gov/rest/json/cves/2.0?keywordSearch=django
I get the response after around 1:40 minutes
and sometimes I just get 503/504 server error response
is it the same issue?

@vehemont
Copy link
Owner

using nvdlib 0.7.4, for around 10 days it is not working for me and now I tested this https://services.nvd.nist.gov/rest/json/cves/2.0?keywordSearch=django I get the response after around 1:40 minutes and sometimes I just get 503/504 server error response is it the same issue?

Yes. That is the issue we are experiencing at this time. Feel free to reach out to the NVD and inform them the issue is still occurring.

@johnlabuyfoy1024
Copy link
Author

Update from NVD on timeout / 503 errors on 7-17-2023:

"We are aware of the root cause for the issue and are working to find appropriate resolutions that will have minimal impact to the userbase while improving reliability of the site and search. We do not have a timeline for this currently and appreciate your patience during this period."

@vehemont
Copy link
Owner

I will push out an update disabling the keyword and keyword exact match options. I will re-enable them once the issue is resolved.

Update from the NVD:

Keyword and Keyword Exact Match Searches Temporarily Disabled
The NVD has been experiencing issues with website and API availability. We have identified the root cause, however, due to the particular complexities and other operational needs, a larger scale solution must be put into place. This will take time to implement and resolve. In the interim, to ensure continuity of services that are not impacted, we will be disabling both the keyword and keyword exact match capabilities of the vulnerability search page and APIs. We are aware that this will impact the daily efforts of many that make use of our data and request understanding and patience while we move towards a viable solution.
For questions and concerns you can contact nvd@nist.gov . Please refrain from requesting timelines on resolution, we will notify all users through the various channels available when we have information to share on the topic.
V/r,
The National Vulnerability Database Team

@mhdawson
Copy link

@vehemont I assume it 0.7.5 that I should wait for in terms of the above?

@vehemont
Copy link
Owner

vehemont commented Aug 3, 2023

NVD released news regarding this issue. NVDLib should be back in functioning order. I don't think any changes will need to be made to NVDLib but I'll do some testing and find out.

Keyword and keyword exact match searches have been re-enabled. Clarifications on how keyword search operates can be found in the documentation for keyword parameters. For questions and concerns you can contact nvd@nist.gov .

@Fares-Harri
Copy link

Fares-Harri commented Aug 6, 2023

Yes it is working now again

@vehemont vehemont closed this as not planned Won't fix, can't repro, duplicate, stale Aug 10, 2023
@mhdawson
Copy link

Seems to be working for the nodejs automation, thanks for this issue and the updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

6 participants