Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to expose elasticsearch port 9200 externally(outside docker) and connect #556

Open
ashishmgupta opened this issue Apr 25, 2021 · 5 comments
Labels
custom build Using the HELK with settings that have not been tested or recommended yet

Comments

@ashishmgupta
Copy link

ashishmgupta commented Apr 25, 2021

Describe the problem

I'm trying to send logs from the Office 365 using the o365 filebeat to the ElasticSearch.
Filebeat gets the data from o365 but not able to send to ElasticSearch and shows below error
Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "http://localhost:9200": EOF
I added 9200:9200 to the docker config files so It could be exposed externally outside docker.
helk-kibana-analysis-alert-basic.yml
helk-kibana-analysis-basic.yml
helk-kibana-notebook-analysis-alert-basic.yml
helk-kibana-notebook-analysis-basic.yml

and then composed docker for each file.
docker-compose -f docker/<config> up --build -d

After composing the docker using one file, I would test the filebeat using
filebeat -e

and It would give me the same error Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get "http://localhost:9200": EOF

Attaching the config files here.
helk-kibana-analysis-alert-basic.yml.txt
helk-kibana-analysis-basic.yml.txt
helk-kibana-notebook-analysis-alert-basic.yml.txt
helk-kibana-notebook-analysis-basic.yml.txt

Provide the output of the following commands

Get operating system and version
for linux (except Mac) use:
cat /etc/os-release
for Mac/OSX use:
sw_vers
Get disk space, memory, processor cores, and docker storage
echo -e "\nDocker Space:" && df -h /var/lib/docker; echo -e "\nMemory:" && free -g; echo -e "\nCores:" && getconf _NPROCESSORS_ONLN
Get output of the HELK docker containers:
docker ps --filter "name=helk"

Place all output, from the above commands, here

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G   24G   24G  50% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:              9           8           0           0           1           0
Swap:             1           1           0

Cores:
4


CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS                          PORTS                                                                                                                                                                                                                          NAMES
6faf14986e88   otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   12 minutes ago   Up 12 minutes                   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                                                                                                       helk-nginx
c862b1efcfdc   otrf/helk-spark-worker:2.4.5                          "./spark-worker-entr…"   13 minutes ago   Up 13 minutes                                                                                                                                                                                                                                                  helk-spark-worker
7519521961bb   docker_helk-jupyter                                   "/opt/jupyter/script…"   13 minutes ago   Up 13 minutes                   8000/tcp, 8888/tcp                                                                                                                                                                                                             helk-jupyter
873c054ac1b3   otrf/helk-spark-master:2.4.5                          "./spark-master-entr…"   13 minutes ago   Up 13 minutes                   7077/tcp, 0.0.0.0:8080->8080/tcp                                                                                                                                                                                               helk-spark-master
272cfe509229   otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   45 minutes ago   Restarting (1) 47 seconds ago                                                                                                                                                                                                                                  helk-kafka-broker
560a76acd0ed   otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   45 minutes ago   Up 45 minutes                   0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:9200->9200/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp   helk-logstash
ccbfa5fb9275   confluentinc/cp-ksql-cli:5.1.3                        "/bin/sh"                2 weeks ago      Up 45 minutes                                                                                                                                                                                                                                                  helk-ksql-cli
5f2a5a34b2dd   confluentinc/cp-ksql-server:5.1.3                     "/etc/confluent/dock…"   2 weeks ago      Up 10 hours                     0.0.0.0:8088->8088/tcp                                                                                                                                                                                                         helk-ksql-server
2941692e1a9a   otrf/helk-elastalert:latest                           "./elastalert-entryp…"   2 weeks ago      Up 10 hours                                                                                                                                                                                                                                                    helk-elastalert
858d52e2c774   otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   2 weeks ago      Up 10 hours                     2181/tcp, 2888/tcp, 3888/tcp                                                                                                                                                                                                   helk-zookeeper
58f1e061c42e   docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   2 weeks ago      Up 10 hours                     5601/tcp                                                                                                                                                                                                                       helk-kibana
d8d753680a0d   docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   2 weeks ago      Up 10 hours                     9200/tcp, 9300/tcp                                                                                                                                                                                                             helk-elasticsearch

What version of HELK are you using

run the command from within the HELK repo run git log -1 --oneline

Place the output here

What version of Winlogbeat are you using if you are using Windows/WEF logs

b40f92f (HEAD -> master, origin/master, origin/HEAD) Update kibana.md
What steps did you take trying to fix the issue
How could we replicate the issue
Any additionally code or log context you would like to provide
Place the output here

Any additional context or input you have

pictures, comments, etc.

@neu5ron
Copy link
Collaborator

neu5ron commented Apr 28, 2021

you can use nginx to "expose" port 9200 to forward to elasticsearch

@ashishmgupta
Copy link
Author

ashishmgupta commented Apr 28, 2021 via email

@Cyb3rWard0g
Copy link
Owner

I just checked one of your configs, and I see you modifying the docker config file and adding port 9200 to it, but you added it to the logstash service and not Elasticsearch. can you verify that please @ashishmgupta ? thank you!

image

@Cyb3rWard0g Cyb3rWard0g added the custom build Using the HELK with settings that have not been tested or recommended yet label May 9, 2021
@ashishmgupta
Copy link
Author

ashishmgupta commented May 9, 2021 via email

@ashishmgupta
Copy link
Author

ashishmgupta commented May 9, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
custom build Using the HELK with settings that have not been tested or recommended yet
Projects
None yet
Development

No branches or pull requests

3 participants