Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐞🐋: Elasticsearch container does not come up/constantly restarts; state always: 'Restarting': sudo docker ps -a | grep elasticsearch #461

Open
1 task done
RogerWeihrauch opened this issue Apr 22, 2024 · 6 comments
Labels
Docker SELKS on Docker

Comments

@RogerWeihrauch
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Module 'Elasticsearch' always greyed-out in 'https://localhost/rules/es'; pls see screen shot:

Screenshot 2024-04-22 at 08-20-48 Scirius Community Edition - Es

Expected Behavior

everything depends on this Elasticsearch container running; so, well, I want this ti start-up and run stable.
So, since this one is not running reliably, also Kibana does not get any useful informations.

Steps To Reproduce

1st) followed this guide to set it up: https://github.com/StamusNetworks/SELKS/wiki/Docker
2nd) after executing: 'sudo -E docker compose up -d'; watch for the Elasticsearch container and its restarts w/ 'docker ps -a | grep -i elasticsearch'

Docker version

selks-user@selks:$ docker -v Docker version 26.0.2, build 3c863ff selks-user@selks:$

Docker version

selks-user@selks:$ docker-compose -v docker-compose version 1.29.2, build 5becea4c selks-user@selks:$

OS Version

selks-user@selks:$ lsb_release -d Description: Debian GNU/Linux 11 (bullseye) selks-user@selks:$

Content of the environnement File

selks-user@selks:/opt/selksd/SELKS/docker$ cat .env
COMPOSE_PROJECT_NAME=selks
INTERFACES= -i enp0s17 -i enp0s8
ELASTIC_DATAPATH=/var/SELKS/ELKdb/
SCIRIUS_SECRET_KEY=MBoZcxs576FcYqh2HEypMSblCG7V5p4YCB0aE8Uo3_M
PWD=${PWD}
selks-user@selks:/opt/selksd/SELKS/docker$

Version of SELKS

selks-user@selks:/opt/selksd/SELKS/docker$ git log -1
commit 2fc5391 (HEAD -> master, origin/master, origin/HEAD)
Merge: a030b9a 16fc908
Author: Eric Leblond eleblond@stamus-networks.com
Date: Mon Sep 11 08:35:37 2023 +0000

Merge branch 'Arkime-fix-v1' into 'master'

Add oui file for Arkime

See merge request devel/SELKS!5

selks-user@selks:/opt/selksd/SELKS/docker$

Anything else?

I am really new to docker/SIEM/SELKS, so I am sure to have done some errors on this.
But:
My assumption on this issue:

  1. http(s)://localhost:9200 cannot be reached (?) within docker environment
    -> so may be an error/misconfig in above selected NICs?
    -> which (v)NIC/vNetwork to select?
    -> logs entries:
    selks-user@selks:/opt/selksd/SELKS/docker/containers-data/scirius/logs$ tail elasticsearch.log
    return func(*args, params=params, **kwargs)
    File "/root/.local/lib/python3.9/site-packages/elasticsearch/client/cluster.py", line 59, in health
    return self.transport.perform_request(
    File "/root/.local/lib/python3.9/site-packages/elasticsearch/transport.py", line 402, in perform_request
    status, headers_response, data = connection.perform_request(
    File "/root/.local/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 245, in perform_request
    raise ConnectionError("N/A", str(e), e)
    elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f30404f5d00>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f30404f5d00>: Failed to establish a new connection: [Errno -2] Name or service not known)

ES connection error: <urllib3.connection.HTTPConnection object at 0x7f30404f5d00>: Failed to establish a new connection: [Errno -2] Name or service not known
selks-user@selks:/opt/selksd/SELKS/docker/containers-data/scirius/logs$

  1. maybe a 'too old' version of Elasticsearch (7.xx) since they are already on 8.xx?
    -> how to upgrade?
    -> how to create an ISO file with those new relases of elastic/kibana/logstash/...?
    -> does the point above make any since here?

  2. ELASTIC_DATAPATH is definitily acessable from selks-user to read/write into it

If you may need any further information on this, pls let me know; I will deliver as fast as possible.

Any hint on this issue is highly appreciated.

Thank you very much for your effort.

Regards,
Roger

@RogerWeihrauch RogerWeihrauch added the Docker SELKS on Docker label Apr 22, 2024
@pevma
Copy link
Member

pevma commented Apr 23, 2024

What is the output of docker ps -a ?
My guess is that it probably needs restarting the Elasticsearch/Logstash containers:

docker compose restart elasticsearch logstash

@RogerWeihrauch
Copy link
Author

RogerWeihrauch commented Apr 23, 2024

@pevma
Hi Peter thanx for answering that fast and sorry for responding that late.
Well, got some more trouble after cold booting up the machine this morning.
So, actual state is:


selks-user@selks:~$ docker ps -a
CONTAINER ID   IMAGE                                        COMMAND                  CREATED          STATUS                          PORTS                                                           NAMES
faf220ef7958   elastic/kibana:7.16.1                        "/bin/tini -- /usr/l…"   9 minutes ago    Up 9 minutes (unhealthy)        5601/tcp                                                        kibana
677d8d53d874   nginx                                        "/docker-entrypoint.…"   9 minutes ago    Up 9 minutes                    80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                   nginx
ac9f5eb0f1d1   ghcr.io/stamusnetworks/scirius:master        "/bin/bash /opt/scir…"   9 minutes ago    Up 9 minutes (unhealthy)        8000/tcp                                                        scirius
17da9aa55b29   jasonish/evebox:master                       "/docker-entrypoint.…"   9 minutes ago    Up 9 minutes                                                                                    evebox
6ff8fa312af3   elastic/elasticsearch:7.16.1                 "/bin/tini -- /usr/l…"   9 minutes ago    Restarting (1) 34 seconds ago                                                                   elasticsearch
dbe63b655374   docker:latest                                "dockerd-entrypoint.…"   9 minutes ago    Up 9 minutes                    2375-2376/tcp                                                   cron
1bad17cc62ef   ghcr.io/stamusnetworks/arkimeviewer:master   "/start-arkimeviewer…"   9 minutes ago    Up 9 minutes                    8005/tcp                                                        arkime
c794c112e9c1   portainer/portainer-ce                       "/portainer --logo h…"   24 minutes ago   Up 12 minutes                   8000/tcp, 9000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp   portainer
selks-user@selks:~$
  1. I assume, kibana and scirius are unhaelthy due to Elasticsearch ist not stable running/in up state and always restarting, isn't it?
  2. I will do a single restart now as you described in your last reponse the following way:
    stop kibana
    stop scirius
    stop elasticsearch, than,always wait until the newly started one is up:
    start elasticsearch, wait,
    start kibana, wait,
    start scirius

What do you think?
OR, is there a special chronology to be respected in re-starting all the tools?
Which are the dependencies of each other?

Regards,
Roger

@RogerWeihrauch
Copy link
Author

Ok, tried to restart elasticsearch container; behavior is still the same:
always restarting and never stable running.
So, what can I do here?

@RogerWeihrauch
Copy link
Author

RogerWeihrauch commented Apr 23, 2024

selks-user@selks:/opt/selksd/SELKS/docker$ docker ps -a
CONTAINER ID   IMAGE                                        COMMAND                  CREATED          STATUS                         PORTS                                                           NAMES
faf220ef7958   elastic/kibana:7.16.1                        "/bin/tini -- /usr/l…"   26 minutes ago   Exited (137) 5 minutes ago                                                                     kibana
677d8d53d874   nginx                                        "/docker-entrypoint.…"   26 minutes ago   Up 26 minutes                  80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                   nginx
ac9f5eb0f1d1   ghcr.io/stamusnetworks/scirius:master        "/bin/bash /opt/scir…"   26 minutes ago   Exited (137) 4 minutes ago                                                                     scirius
17da9aa55b29   jasonish/evebox:master                       "/docker-entrypoint.…"   26 minutes ago   Up 26 minutes                                                                                  evebox
6ff8fa312af3   elastic/elasticsearch:7.16.1                 "/bin/tini -- /usr/l…"   26 minutes ago   Restarting (1) 9 seconds ago                                                                   elasticsearch
dbe63b655374   docker:latest                                "dockerd-entrypoint.…"   26 minutes ago   Up 26 minutes                  2375-2376/tcp                                                   cron
1bad17cc62ef   ghcr.io/stamusnetworks/arkimeviewer:master   "/start-arkimeviewer…"   26 minutes ago   Up 26 minutes                  8005/tcp                                                        arkime
c794c112e9c1   portainer/portainer-ce                       "/portainer --logo h…"   41 minutes ago   Up 29 minutes                  8000/tcp, 9000/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp   portainer
selks-user@selks:/opt/selksd/SELKS/docker$

@RogerWeihrauch
Copy link
Author

RogerWeihrauch commented Apr 23, 2024

selks-user@selks:/opt/selksd/SELKS/docker/containers-data/scirius/logs$ tail elasticsearch.log 
    return func(*args, params=params, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 285, in exists
    return self.transport.perform_request("HEAD", _make_path(index), params=params)
  File "/root/.local/lib/python3.9/site-packages/elasticsearch/transport.py", line 402, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/root/.local/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 245, in perform_request
    raise ConnectionError("N/A", str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4cb14944c0>: Failed to resolve 'elasticsearch' ([Errno -2] Name or service not known)) caused by: NameResolutionError(<urllib3.connection.HTTPConnection object at 0x7f4cb14944c0>: Failed to resolve 'elasticsearch' ([Errno -2] Name or service not known))

ES connection error: <urllib3.connection.HTTPConnection object at 0x7f4cb14944c0>: Failed to resolve 'elasticsearch' ([Errno -2] Name or service not known)
selks-user@selks:/opt/selksd/SELKS/docker/containers-data/scirius/logs$ 

@pevma
Copy link
Member

pevma commented Apr 24, 2024

Could please verify, if you have at least 2 CPUs and 10GB RAM on the host.
Then restart just the scirius/kibana containers:

docker compose restart scirius kibana

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docker SELKS on Docker
Projects
None yet
Development

No branches or pull requests

2 participants