Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CVE and CPE Imports and Updates fail #1092

Closed
derIckeBrln opened this issue May 15, 2024 · 2 comments
Closed

CVE and CPE Imports and Updates fail #1092

derIckeBrln opened this issue May 15, 2024 · 2 comments

Comments

@derIckeBrln
Copy link

derIckeBrln commented May 15, 2024 •
edited

on initial db population i receive, there are seem to be no CPE and CVE:

./cve-search/sbin/db_updater.py -f

2024-05-15 09:38:05,117 - CveXplore.main - INFO - Using mongodb as datasource, connection details: None
2024-05-15 09:38:05,124 - CveXplore.main - INFO - Initialized CveXplore version: 0.3.31
2024-05-15 09:38:05,124 - DBUpdater - INFO - ==========================
2024-05-15 09:38:05,124 - DBUpdater - INFO - Repopulate
2024-05-15 09:38:05,124 - DBUpdater - INFO - Wed 15 May 2024 07:38
2024-05-15 09:38:05,124 - DBUpdater - INFO - ==========================
2024-05-15 09:38:05,124 - DBUpdater - INFO - Dropping metadata: cpeother
2024-05-15 09:38:05,128 - DBUpdater - INFO - Dropping metadata: mgmt_whitelist
2024-05-15 09:38:05,130 - DBUpdater - INFO - Dropping metadata: mgmt_blacklist
2024-05-15 09:38:05,134 - DBUpdater - INFO - Dropping metadata: info
2024-05-15 09:38:05,135 - DBUpdater - INFO - Dropping metadata: schema
2024-05-15 09:38:05,137 - DBUpdater - INFO - Starting initial import...
2024-05-15 09:38:05,137 - CveXplore.core.database_maintenance.main_updater - INFO - Starting Database initialization....
2024-05-15 09:38:05,143 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:38:05,144 - CveXplore.core.database_maintenance.sources_process - INFO - CPE Database population started
2024-05-15 09:38:05,154 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
2024-05-15 09:38:15,592 - CveXplore.core.database_maintenance.sources_process - INFO - Preparing to download 0 CPE entries
Downloading and processing content: 0it [00:11, ?it/s]
2024-05-15 09:38:27,185 - CveXplore.core.database_maintenance.sources_process - INFO - Duration: 0:00:22.031330
2024-05-15 09:38:27,200 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('id', 1)] on cpe
2024-05-15 09:38:27,207 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vendor', 1)] on cpe
2024-05-15 09:38:27,212 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('product', 1)] on cpe
2024-05-15 09:38:27,221 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('deprecated', 1)] on cpe
2024-05-15 09:38:27,229 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('cpeName', 1)] on cpe
2024-05-15 09:38:27,235 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('title', 1)] on cpe
2024-05-15 09:38:27,242 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('stem', 1)] on cpe
2024-05-15 09:38:27,248 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('padded_version', 1)] on cpe
2024-05-15 09:38:27,255 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('lastModified', 1)] on cpe
2024-05-15 09:38:27,255 - CveXplore.core.database_maintenance.sources_process - INFO - Finished CPE database population
2024-05-15 09:38:27,255 - CveXplore.core.database_maintenance.main_updater - INFO - Sleeping for 30 seconds between CPE and CVE database population..
2024-05-15 09:38:57,286 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:38:57,287 - CveXplore.core.database_maintenance.sources_process - INFO - CVE database population started
2024-05-15 09:38:57,287 - CveXplore.core.database_maintenance.sources_process - INFO - Starting CVE database population starting from year: 2000
2024-05-15 09:38:57,309 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
2024-05-15 09:39:09,111 - CveXplore.core.database_maintenance.sources_process - INFO - Preparing to download 0 CVE entries
Downloading and processing content: 0it [00:13, ?it/s]
2024-05-15 09:39:22,261 - CveXplore.core.database_maintenance.sources_process - INFO - Duration: 0:00:24.951769
2024-05-15 09:39:22,284 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('id', 1)] on cves
2024-05-15 09:39:22,291 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vulnerable_configuration', 1)] on cves
2024-05-15 09:39:22,310 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vulnerable_product', 1)] on cves
2024-05-15 09:39:22,316 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('modified', 1)] on cves
2024-05-15 09:39:22,324 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('published', 1)] on cves
2024-05-15 09:39:22,332 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('lastModified', 1)] on cves
2024-05-15 09:39:22,340 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('cvss', 1)] on cves
2024-05-15 09:39:22,349 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('cvss3', 1)] on cves
2024-05-15 09:39:22,356 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('summary', 'text')] on cves
2024-05-15 09:39:22,364 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vendors', 1)] on cves
2024-05-15 09:39:22,371 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('products', 1)] on cves
2024-05-15 09:39:22,379 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('assigner', 1)] on cves
2024-05-15 09:39:22,385 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('cwe', 1)] on cves
2024-05-15 09:39:22,393 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('status', 1)] on cves
2024-05-15 09:39:22,401 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vulnerable_product_stems', 1)] on cves
2024-05-15 09:39:22,409 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2024-05-15 09:39:22,418 - CveXplore.core.database_indexer.db_indexer - INFO - Success to create index [('epss', 1)] on cves
2024-05-15 09:39:22,418 - CveXplore.core.database_maintenance.sources_process - INFO - Finished CVE database population
2024-05-15 09:39:22,419 - CveXplore.core.database_maintenance.main_updater - INFO - Starting Database update....
2024-05-15 09:39:22,428 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:39:22,429 - CveXplore.core.database_maintenance.sources_process - INFO - CPE database update started
2024-05-15 09:39:22,429 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
2024-05-15 09:39:22,431 - CveXplore.core.database_maintenance.sources_process - WARNING - No records found in the mongodb cpe collection..
2024-05-15 09:39:22,432 - CveXplore.core.database_maintenance.sources_process - INFO - Finished CPE database update
2024-05-15 09:39:22,441 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:39:22,441 - CveXplore.core.database_maintenance.sources_process - INFO - CVE database update started
2024-05-15 09:39:22,442 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
2024-05-15 09:39:22,443 - CveXplore.core.database_maintenance.sources_process - WARNING - No records found in the mongodb cves collection..
2024-05-15 09:39:22,444 - CveXplore.core.database_maintenance.sources_process - INFO - Finished CVE database update

CWE and CAPEC work fine.
After the initial creation the update fails as well:

./cve-search/sbin/db_updater.py -v
2024-05-15 09:52:37,799 - CveXplore.main - INFO - Using mongodb as datasource, connection details: None
2024-05-15 09:52:37,806 - CveXplore.main - INFO - Initialized CveXplore version: 0.3.31
2024-05-15 09:52:37,806 - DBUpdater - INFO - ==========================
2024-05-15 09:52:37,806 - DBUpdater - INFO - Update
2024-05-15 09:52:37,806 - DBUpdater - INFO - Wed 15 May 2024 07:52
2024-05-15 09:52:37,806 - DBUpdater - INFO - ==========================
2024-05-15 09:52:37,806 - CveXplore.core.database_maintenance.main_updater - INFO - Starting Database update....
2024-05-15 09:52:37,812 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:52:37,813 - CveXplore.core.database_maintenance.sources_process - INFO - CPE database update started
2024-05-15 09:52:37,813 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
2024-05-15 09:52:37,814 - CveXplore.core.database_maintenance.sources_process - WARNING - No records found in the mongodb cpe collection..
2024-05-15 09:52:37,815 - CveXplore.core.database_maintenance.sources_process - INFO - Finished CPE database update
2024-05-15 09:52:37,820 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - NVD NIST API Key found!
2024-05-15 09:52:37,820 - CveXplore.core.database_maintenance.sources_process - INFO - CVE database update started
2024-05-15 09:52:37,821 - CveXplore.core.database_maintenance.sources_process - INFO - Starting download...
Traceback (most recent call last):
File "/virtimo/cve-search/sbin/db_updater.py", line 215, in
main(args)
File "/virtimo/cve-search/sbin/db_updater.py", line 99, in main
cvex.database.update()
File "/virtimo/.local/lib/python3.10/site-packages/CveXplore/core/database_maintenance/main_updater.py", line 96, in update
up.update()
File "/virtimo/.local/lib/python3.10/site-packages/CveXplore/core/database_maintenance/sources_process.py", line 776, in update
self.process_downloads()
File "/virtimo/.local/lib/python3.10/site-packages/CveXplore/core/database_maintenance/sources_process.py", line 710, in process_downloads
raise KeyError(
KeyError: "Missing field 'lastModified' from database query..."

I can't find any details in ~/.cvexplore/log/cvexplore.log
The NIST API key is set
Furthermore I can curl https://nvd.nist.gov/feeds/json/cve/1.1/
Looking into code this may be due to "except ApiMaxRetryError:" but I can't manage to identify the API Error
@derIckeBrln
Copy link
Author

I just implemented some debug logging into nvd_nist_api.py and api_base_class.py and currently can see, that the URL is built correctly but we seem to face certain 404:

2024-05-21 12:23:31,559 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Making API call with method: GET, RESSOURCE: {'resultsPerPage': 1}
2024-05-21 12:23:31,559 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Built URL: https://services.nvd.nist.gov/rest/json/cpes/2.0/?resultsPerPage=1
2024-05-21 12:23:31,560 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Sending GET request to URL: https://services.nvd.nist.gov/rest/json/cpes/2.0/?resultsPerPage=1
2024-05-21 12:23:31,560 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Request headers: {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36', 'apiKey': '************************'}
2024-05-21 12:23:31,560 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Request data: 2
2024-05-21 12:23:55,366 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Received response: 404
2024-05-21 12:23:55,367 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Response content:
2024-05-21 12:23:55,367 - CveXplore.core.nvd_nist.nvd_nist_api - INFO - Received API response: <Response [404]>

anyone else facing issues like that currently?
Any way around that e.g. by processing the zips?

@derIckeBrln
Copy link
Author

Okay, managed to make it work... NIST seems to create non working API keys.
A second one did not work as well..
Just the third created key from NIST made the API work again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@derIckeBrln