Queued Sessions remain in PENDING state even after the host is back online

[Rajeswara-Muthyala]

I'm using the sample script from the falconpy git repo to execute commands.
https://github.com/CrowdStrike/falconpy/blob/3edfb2b5fb98fc939937942369c58d53d8223cd1/samples/rtr/queued_execute.py.

For an offline host, the commands are getting queued on the session as expected. However, in most of my experiments, the sessions remain in PENDING even after bringing up the host online. I waited for days after bring the host online, but the session remains in PENDING state. I confirmed that the host is online by running devices API (/devices/entities/devices/v2)

Appreciate any guidance on helping to solve this problem.

For what its worth, less frequently, queued sessions were succesfully finished. I couldn't find any pattern. The behavior is unpredictable.

Pasting list_queued_sessions response below

        "api_call": "list_queued_sessions",
        "response": {
            "body": {
                "meta": {
                    "trace_id": <<REDACTED>>,
                    "powered_by": "empower-api",
                    "query_time": 0.004938161
                },
                "errors": {},
                "resources": [
                    {
                        "id": <<REDACTED>>,
                        "aid": <<REDACTED>>,
                        **"status": "PENDING",**
                        "user_id": <<REDACTED>>,
                        "Commands": [
                            {
                                **"status": "INITIALIZED",**
                                "created_at": "2024-01-31T15:48:20Z",
                                "deleted_at": null,
                                "updated_at": "2024-01-31T15:48:20Z",
                                "base_command": "runscript",
                                "command_string": <<REDACTED>>
                            },
                            {
                                **"status": "INITIALIZED",**
                                "created_at": "2024-01-31T15:48:21Z",
                                "deleted_at": null,
                                "updated_at": "2024-01-31T15:48:21Z",
                                "base_command": "runscript",
                                "command_string": <<REDACTED>>
                            },
                            {
                                **"status": "INITIALIZED",**
                                "created_at": "2024-01-31T15:48:21Z",
                                "deleted_at": null,
                                "updated_at": "2024-01-31T15:48:21Z",
                                "base_command": "runscript",
                                "command_string": <<REDACTED>>
                            }
                        ],
                        "user_uuid": "",
                        "created_at": "2024-01-31T15:48:16Z",
                        "deleted_at": null,
                        "updated_at": "2024-01-31T15:48:16Z"
                    }
                ]
            },
            "headers": {
                **"Date": "Thu, 01 Feb 2024 07:05:33 GMT",**
                "Server": "nginx",
                "Connection": "keep-alive",
                "X-Cs-Region": "us-1",
                "Content-Type": "application/json",
                "X-Cs-Traceid": <<REDACTED>>,
                "Content-Length": "1009",
                "Content-Encoding": "gzip",
                "X-Ratelimit-Limit": "6000",
                "X-Ratelimit-Remaining": "5839",
                "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
            },
            "status_code": 200
        }

[jshcodes]

It will take a few minutes from the time the host is taken down to when they will show up as "offline" within the console. During this period, commands would be issued to the host (and then time out) as opposed to being queued.

For the experimental hosts that remain at pending, were these hosts taken down within minutes of the queued commands being sent?

[Rajeswara-Muthyala]

Thanks for the response.

I understand the behaviour that the commands will be timed out if they are issued within minutes of bringing the host offline. Instead, my experiment is a more straightforward and simple one.

In my experiments, I even tried issuing commands hours after the hosts are brought offline. APIs are indeed reporting that the hosts are offline and so the commands are queued. But, when i bring these hosts online, these commands won't get executed. They will simply remain in pending state (for hours/days).

FWIW, in my experiments, i'm using Windows VMs with CS agents. All online RTR sessions (commands issued when the hosts are online) are running as expected. Also, one related observation is that, the problem i reported, isn't happening when the VM is suspended and brought back online. It's more often reproducible when the VM is shutdown and brought back online.

Appreciate your help. Kindly let us know if you need any more details.

[jshcodes]

Hi @Rajeswara-Muthyala - We are working on recreating this, stay tuned.

[jshcodes]

@Rajeswara-Muthyala - Would you be willing to open a support ticket and reference this GitHub discussion? The support team will assign the ticket to us and we will reach out directly. We're not recreating this issue reliably, and would like more detail.

[Rajeswara-Muthyala]

Sure