Operation halts with asyncio timeout exception in logs?

[calebshortt]

Hi all,

I have run into an issue where a Caldera operation will halt indefinitely. When I check the logs (via journalctl) I see a "asyncio.exceptions.TimeoutError" that is getting thrown on "File "/home/atadmin/caldera/app/service/event_svc.py", line 83, in fire_event". It is breaking on the websocket.connect method: "async with websockets.connect(uri) as websocket:".
This is happening regularly.

Quick Note: I am running on an Azure instance (Standard_D4s_v3: 16GB, 4 CPUs); Ubuntu 20.04 LTS; Caldera install is on the system itself and not using docker.

Some investigation and thoughts:

• Gut feeling: This smells of a resource constraint especially considering that the websocket timeouts are going to localhost...aka: there shouldn't be any network lag on this one (Though I am unsure if letting such an exception as this halt entire operations instead of gracefully handling them is a good idea either?).
• When I run "mpstat -P ALL 2 3" to get a feel for what is happening, I see that Caldera is pinning one of the 4 CPUs to 100%. I have tried to find config information on this, to make Caldera use all CPUs, but could not. Am I missing something? I will be attempting the docker install in an attempt to make this work across all CPUs.
• In my investigation, I also thought that Caldera might be burning through websockets faster than the timeouts could free them (though it would have to be hitting a lot of websockets to do so); when I use "netstat -np" to watch the connections, there are hardly any -- though I can see in the source (https://github.com/mitre/caldera/blob/master/app/service/event_svc.py#L83) that it is creating a new websocket for this purpose then killing it (using the with keyword), so Caldera could be burning through websockets and closing them -- A possible improvement here could be websocket re-use, but that may be out of scope for this discussion). Suffice it to say, This could be contributing to the issue.

My REAL questions are...

• Has anyone encountered this before?
• Am I missing a critical aspect of the config where I set Caldera to use all CPUs?

UPDATE

• I have upgraded from Azure's "Standard_D4s_v3" (4vCPUs, 16GB) to "Standard_D8s_v3" (8 vCPUs, 32GB) (https://learn.microsoft.com/en-us/azure/virtual-machines/dv3-dsv3-series) and the problem persists.
• Running "mpstat -P ALL 2 10" I am seeing that the CPU core ARE being used, one at a time, and each one that is being used are pinned to near-100%.

Thanks for the help

Logs:

[calebshortt]

I believe I have found the issue and will post here to help any others who may run into the same issue.

I am running Caldera against a target host that has non-blocking EP.
When the Caldera agent (sandcat in this case) runs through its battery of tests (i.e. operation), the host gets pinned at 100% CPU usage (the server being pinned at 100% usage seems to be a red herring). This is causing the websockets to time out because of this line:

https://github.com/mitre/caldera/blob/master/app/service/event_svc.py#L83

Which calls the websocket.connect(uri) command with default timeout (which is 10s).
Unfortunately I am unable to rectify the high CPU usage on the host, so my fix was to change the "connect" call above to "websocket.connect(uri, open_timeout=60)" to give it a 60s timeout instead of 10s -- as per the websocket docs: https://websockets.readthedocs.io/en/stable/reference/client.html#websockets.client.connect

Not a "good" fix, but Caldera was crashing every time an operation was run without it.
Hope this helps someone else.