Grid Page Empty After Node Update

[not-mike-wazowski]

Version

2.4.60

Installation Method

Other (please provide detail below)

Description

upgrading

Installation Type

Standalone

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

255G

Storage for /nsm

255G

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I've been stably running SO via the Azure Marketplace image for a couple weeks now. Today, I logged in and noticed on the Grid page my sole node (standalone mode) status bar was Red. Clicking around it seemed SO wanted me to update the node. I obliged by clicking the update button in the UI, and after it completed I can no longer see the node (or any info) in the Grid page.

so-status shows all services running correctly, and I can view my Crowdstrike-integrated logs in both SOC as well as Elastic, so I don't believe there's anything severe going on here. That being said, I'd love to get the bottom of why this happened. I tried a full reset using so-elastic-fleet-reset, which completed successfully with no change to the Grid page. I did find these logs which seem relevant:

[admin@securityonion ~]$ sudo tail /opt/so/log/elasticfleet/elastic-agent-20240429-3.ndjson | grep error | jq
{
  "log.level": "error",
  "@timestamp": "2024-04-30T00:38:36.168Z",
  "message": "Error dialing dial tcp 127.0.0.1:9200: connect: connection refused",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "service.name": "filebeat",
  "address": "localhost:9200",
  "ecs.version": "1.6.0",
  "log.logger": "esclientleg",
  "log.origin": {
    "file.line": 38,
    "file.name": "transport/logging.go"
  },
  "network": "tcp"
}
{
  "log.level": "error",
  "@timestamp": "2024-04-30T00:39:21.739Z",
  "message": "Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get \"http://localhost:9200\": dial tcp 127.0.0.1:9200: connect: connection refused",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "log.logger": "publisher_pipeline_output",
  "log.origin": {
    "file.line": 148,
    "file.name": "pipeline/client_worker.go"
  }
}
{
  "log.level": "error",
  "@timestamp": "2024-04-30T00:39:21.740Z",
  "message": "Error dialing dial tcp 127.0.0.1:9200: connect: connection refused",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "esclientleg",
  "service.name": "filebeat",
  "ecs.version": "1.6.0",
  "network": "tcp",
  "address": "localhost:9200",
  "log.origin": {
    "file.line": 38,
    "file.name": "transport/logging.go"
  }
}
{
  "log.level": "error",
  "@timestamp": "2024-04-30T00:40:05.826Z",
  "message": "Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get \"http://localhost:9200\": dial tcp 127.0.0.1:9200: connect: connection refused",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "publisher_pipeline_output",
  "log.origin": {
    "file.line": 148,
    "file.name": "pipeline/client_worker.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}
{
  "log.level": "error",
  "@timestamp": "2024-04-30T00:40:05.826Z",
  "message": "Error dialing dial tcp [::1]:9200: connect: cannot assign requested address",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.origin": {
    "file.line": 38,
    "file.name": "transport/logging.go"
  },
  "network": "tcp",
  "address": "localhost:9200",
  "ecs.version": "1.6.0",
  "log.logger": "esclientleg",
  "service.name": "filebeat"
}

Any idea what the problem might be or where I should look next? Thanks!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[not-mike-wazowski]

Oh, I'm also seeing the same error described here: #11207 in my logstash logs

[not-mike-wazowski]

bump

This is becoming an issue as I cannot view agent utilization metrics, which is one of our primary use cases for deploying SO

[not-mike-wazowski]

Update: found another error in /opt/so/log/elasticsearch/securityonion.log whic hseems related:

[2024-05-09T00:00:28,318][WARN ][rest.suppressed          ] path: /.fleet-actions/_fleet/global_checkpoints, params: {wait_for_advance=true, wait     _for_index=true, checkpoints=3, index=.fleet-actions, timeout=240000ms}
org.elasticsearch.transport.SendRequestTransportException: [my-securityonion][10.1.1.1:9300][indices:monitor/fleet/global_checkpoints[s][s]]
        at org.elasticsearch.bootstrap.Elasticsearch.shutdown(Elasticsearch.java:467) ~[elasticsearch-8.10.4.jar:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.node.NodeClosedException: node closed {my-securityonion}{CAQag8gbRLmFXHUEs9TFmw}{YQx5scvTQLOaE0ejAzRAZg}{my-securityonion}     {my-securityonion}{10.10.4.4:9300}{dhimrt}{8.10.4}{7000099-8100499}{transform.config_version=10.0.0, xpack.installed=true}
         ... 11 more
[2024-05-09T00:00:28,318][ERROR][io.netty.util.concurrent.DefaultPromise.rejectedExecution] Failed to submit a listener notification task. Event      loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated

[reyesj2]

After running the reset script all agents you have deployed will need to be uninstalled and redeployed. Have you uninstalled and reinstalled the agent on your endpoints?

[not-mike-wazowski]

Just did using "Unenroll agent" and "Add agent". It's back up and Healthy, but still no luck in Grid page. The securityonion.log ERROR is gone. There are lots of warnings like the following:

[2024-05-10T17:35:17,008][WARN ][org.elasticsearch.http.netty4.Netty4HttpServerTransport] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.17.1.22:9200, remoteAddress=/172.17.1.1:52012}

Think it's related?

[not-mike-wazowski]

Tried a fresh install to no avail :/

https://myonion.dev/api/grid returns an empty list []

Everything else seems to be working fine. No clue why I can't get this endpoint to work