vfense server constantly using 100% CPU

[ace008]

Hi,

I have a vfense install with about 100 agents on it. Recently, the CPU on the vfense server is constantly at 100% used. Top shows thats python is constantly pegging the CPU, followed by nginx. Some modules in the web interface are getting errors and cannot load. If I stop the vfense server, the disk IO on the agents becomes really high. Any ideas where to start troubleshooting?

Thanks.

[andrboot]

Hi,
That kind of indicates your hitting the max load on your current server, are you able to bump up your Processing power?

[ace008]

This server already has 4 CPUs assigned to it. How many CPUs do people usually recommend for this?

[andrboot]

Ace, all it all depends how many systems are hitting it, what kind of updates are getting pulled down etc.
How often they are hitting the system.

[ace008]

What's the default schedule for it to scan for updates? There's roughly 100 ubuntu agents pointed to it.

[linuxdynasty]

@ace008 This may mean that the vFense server needs to be scaled up with CPU and DiskIO. When I have seen this, it is due to RethinkDB being pinned.

[ace008]

The web interface no longer loads. Top shows about 10 instances of python running and its taking up all the CPU. I've increased from 4 CPUs to 6 CPUs and still it has the same issue. If I tail the rvweb_file.log I get a lot of these messages:

2016-01-28 10:33:20 - rvweb - ERROR - 500 PUT /rvl/v1/2c99fb22-4939-4d5c-932e-631bb77811ab/rv/updatesapplications (127.0.0.1) 10.99.16.182 10.99.16.182 python-requests/1.2.3 CPython/2.7.3 Linux/3.2.0-54-virtual, 72.41ms

I've tried restarting nginx, redis, rethinkdb to no avail. Any where else I can start troubleshooting?

Thanks.

[linuxdynasty]

This would mean that the server is receiving updates from the clients and its queue is backed up. Stop the vFense app and clean out the data in redis, so you have a fresh queue.

[ace008]

How do I clear out the data in redis?

[linuxdynasty]

redis-cli flushall

[ace008]

Great, looks like the UI is back up. Still seeing alot of CPU usage from Python. I've upped the CPUs to 8 cores. Should I just let it run for the updates to catch up? If I start clicking on other tabs, rethinkDB starts using alot of CPU as well and the UI starts to hang. Anything else to watch out for?

[linuxdynasty]

What is the backend disk. RethinkDB is best ran on SSD Disks, so I would highly recommend, that you put at least RethinkDB on faster volume

[ace008]

The VM is on a Netapp using SAS spinning disks. Any ideas why the app would be using so much CPU? Are there any logs I can check? The app used to work great. Not sure what happened. We have about 100 agents. The UI is so slow currently its unusable.

[linuxdynasty]

Can you see if it is RethinkDB that is stuck on IO? Python is the one making the calls, but RethinkDB is where the queries are happening at. Also are these linux agents?

[ace008]

Correct these are all Ubuntu agents. RAM is all being used up at 12GB as well. Should I increase to 16GB?

[linuxdynasty]

I would definitely increase the ram to 16G. How much of the memory is being used by RethinkDB?