Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting started with mysql storage, batch processing is taking forever #6647

Open
kelfink opened this issue May 6, 2024 · 0 comments
Open
Labels

Comments

@kelfink
Copy link

kelfink commented May 6, 2024

I've set up the NVDAPI key in our projects and that helps a lot, but wanted to incorporate the database storage for caching.
I'm using a MariaDB backend, and found that specifying "character set utf8" in initialize_mysql.sql when creating the database, was needed to avoid DB errors when importing the NVD data.
Now that I'm past that, the "Downloaded ..." logs completed but now I see

[INFO] Completed processing batch 1/125 (1%) in 992,366ms
[INFO] Completed processing batch 2/125 (2%) in 1,784,160ms
...
[INFO] Completed processing batch 14/125 (11%) in 2,254,533ms
[INFO] Completed processing batch 15/125 (12%) in 2,280,112ms
[INFO] Completed processing batch 16/125 (13%) in 2,454,196ms

Here is seems to have gotten stuck. I haven't seen process 17/125.

which seems like an incredibly long time to wait. Is this the database taking a long time? My slow query log doesn't show any queries over 1.5 seconds, so it seems not.... I'm not even sure what 'Completed processing batch' means. Help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant