Skip to content
This repository has been archived by the owner on Apr 12, 2022. It is now read-only.

Hanging on setup #55

Closed
vanosg opened this issue Sep 2, 2018 · 14 comments · May be fixed by #103
Closed

Hanging on setup #55

vanosg opened this issue Sep 2, 2018 · 14 comments · May be fixed by #103

Comments

@vanosg
Copy link

vanosg commented Sep 2, 2018

Hi, on two different machines, docker-compose -f setup.yml up has hung at this point:

Creating elasticsearch ... done
Creating setup_logstash ... done
Creating setup_kibana   ... done
Attaching to setup_logstash, setup_kibana
setup_1  | setup_kibana           | -rw-rw-r-- 1 root root 1200 Sep  2 22:05 /usr/share/kibana/config/ca/ca.crt
setup_1  | setup_logstash         | -rw-rw-r-- 1 root root 1200 Sep  2 22:05 /usr/share/logstash/config/ca/ca.crt

Is the process done here, and I have to manually tear down the containers? Or have they truly hung? Host box is ubuntu 18.01. Please let me know if I can provide anything else, thanks!
@fxdgear
Copy link
Contributor

fxdgear commented Sep 2, 2018

can you share the output of the elasticsearch container?

@vanosg
Copy link
Author

vanosg commented Sep 3, 2018 

<snip>
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key  
setup_1  | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1  | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1  | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1  | setup_elasticsearch exited with code 0

Then it goes into what I pasted before...

setup_1  | setup_elasticsearch exited with code 0
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting elasticsearch ... done
Creating setup_kibana   ... done
Creating setup_logstash ... done
Attaching to setup_kibana, setup_logstash
setup_1  | setup_kibana           | -rw-rw-r-- 1 root root 1200 Sep  2 22:44 /usr/share/kibana/config/ca/ca.crt
setup_1  | setup_logstash         | -rw-rw-r-- 1 root root 1200 Sep  2 22:44 /usr/share/logstash/config/ca/ca.crt

@vanosg
Copy link
Author

vanosg commented Sep 3, 2018

If it helps,

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker-compose version 1.22.0, build f46880fe

@vanosg
Copy link
Author

vanosg commented Sep 3, 2018

I just tried this a fifth time, and it appears to have worked... I don't think I did anything different (but I did re-clone the dir, just to start fresh). Sorry to false-alarm on this

@vanosg vanosg closed this as completed Sep 3, 2018
@fxdgear
Copy link
Contributor

fxdgear commented Sep 3, 2018 via email

@Crysalist
Copy link

I am getting the same issue.

  • Downloaded the ZIP from the repo
  • Extracted it.
  • Installed newest docker-compose (v1.22.0)
  • Did the -setup command
  • hanging at :
    530ed487ab9_stack-docker-master_setup_1 | setup_kibana | -rw-rw-r-- 1 root root 1200 Sep 14 05:50 /usr/share/kibana/config/ca/ca.crt d530ed487ab9_stack-docker-master_setup_1 | setup_logstash | -rw-rw-r-- 1 root root 1200 Sep 14 05:50 /usr/share/logstash/config/ca/ca.crt

Then also

  • deleted all docker images
  • deleted all files
  • rinse and repeat of the above but still no luck

@Crysalist
Copy link

Seems the command :
sysctl -w vm.max_map_count=262144
did not take the first time around. Redid it also and now it works.

@rajiff
Copy link

rajiff commented Oct 7, 2018

I run it on mac, i did below to move past, hope it helps others

docker-machine ssh
sudo sysctl -w vm.max_map_count=262144

@ohing504
Copy link

ohing504 commented Nov 26, 2018 •
edited

In my case, sysctl -w vm.max_map_count=262144 command not help.
I tried docker-compose -f setup.yml up multiple times, and it works rarely.

And I found out the codes on hanging in following codes:

stack-docker/scripts/setup-kibana.sh

Lines 18 to 20 in f9d13bd

while [[ "$(curl -u "elastic:${ELASTIC_PASSWORD}" --cacert $cacert -s -o /dev/null -w '%{http_code}' $es_url)" != "200" ]]; do
sleep 5
done

stack-docker/scripts/setup-logstash.sh

Lines 14 to 18 in f9d13bd

es_url=https://elasticsearch:9200
# Wait for Elasticsearch to start up before doing anything.
while [[ "$(curl -u "elastic:${ELASTIC_PASSWORD}" --cacert $cacert -s -o /dev/null -w '%{http_code}' $es_url)" != "200" ]]; do
sleep 5
done

My logs show Starting elasticsearch ... done but the response of above code is 000.
So, the setup scripts fall asleep...

$ ~/stack-docker$ docker-compose -f setup.yml up
Recreating stack-docker_setup_1_1f2213838db8 ... done
Attaching to stack-docker_setup_1_1f2213838db8
setup_1_1f2213838db8 | Found orphan containers (stack-docker_setup_1_1f2213838db8) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating setup_elasticsearch ... done
Attaching to setup_elasticsearch
setup_1_1f2213838db8 | setup_elasticsearch    | Determining if x-pack is installed...
setup_1_1f2213838db8 | setup_elasticsearch    | === CREATE Keystore ===
setup_1_1f2213838db8 | setup_elasticsearch    | Elastic password is: EhS5NuYLmy3yre9ief4HRg==
setup_1_1f2213838db8 | setup_elasticsearch    | Created elasticsearch keystore in /usr/share/elasticsearch/config
setup_1_1f2213838db8 | setup_elasticsearch    | Setting bootstrap.password...
setup_1_1f2213838db8 | setup_elasticsearch    | === CREATE SSL CERTS ===
setup_1_1f2213838db8 | setup_elasticsearch    | Remove old ca zip...
setup_1_1f2213838db8 | setup_elasticsearch    | Creating docker-cluster-ca.zip...
setup_1_1f2213838db8 | setup_elasticsearch    | CA directory exists, removing...
setup_1_1f2213838db8 | setup_elasticsearch    | Unzip ca files...
setup_1_1f2213838db8 | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster-ca.zip
setup_1_1f2213838db8 | setup_elasticsearch    |    creating: /config/ssl/ca/
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.crt
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.key
setup_1_1f2213838db8 | setup_elasticsearch    | Remove old docker-cluster.zip zip...
setup_1_1f2213838db8 | setup_elasticsearch    | Create cluster certs zipfile...
setup_1_1f2213838db8 | setup_elasticsearch    | Unzipping cluster certs zipfile...
setup_1_1f2213838db8 | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster.zip
setup_1_1f2213838db8 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key
setup_1_1f2213838db8 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key
setup_1_1f2213838db8 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt
setup_1_1f2213838db8 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key
setup_1_1f2213838db8 | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1_1f2213838db8 | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1_1f2213838db8 | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1_1f2213838db8 | setup_elasticsearch exited with code 0
setup_1_1f2213838db8 | Found orphan containers (stack-docker_setup_1_1f2213838db8) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting elasticsearch ... done
Creating setup_logstash ... done
Creating setup_kibana   ... done
Attaching to setup_kibana, setup_logstash
setup_1_1f2213838db8 | setup_kibana           | -rw-rw-r-- 1 root root 1200 Nov 26 09:03 /usr/share/kibana/config/ca/ca.crt
setup_1_1f2213838db8 | setup_logstash         | -rw-rw-r-- 1 root root 1200 Nov 26 09:03 /usr/share/logstash/config/ca/ca.crt

My docker list:

$ docker ps
CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                NAMES
ef0bee2964c4        docker.elastic.co/logstash/logstash:6.3.0   "/usr/local/bin/dock…"   5 minutes ago       Up 4 minutes        5044/tcp, 9600/tcp   setup_logstash
0f579245ad17        docker.elastic.co/kibana/kibana:6.3.0       "/bin/bash -c 'cat /…"   5 minutes ago       Up 5 minutes        5601/tcp             setup_kibana
7622d92acdc3        docker/compose:1.21.2                       "/bin/ash -c 'cat ./…"   5 minutes ago       Up 5 minutes                             stack-docker_setup_1_1f2213838db8

@fxdgear
Copy link
Contributor

fxdgear commented Nov 26, 2018

@ohing504

Can you please check the output of the elasticsearch container.

It looks like elasticsearch is failing to start for some reason.

@minhtd1981
Copy link

@ohing504

Can you please check the output of the elasticsearch container.

It looks like elasticsearch is failing to start for some reason.

I think I got the same issue. Here is the log:

[WARN ][o.e.h.n.Netty4HttpServerTransport] [STIiuSf] caught exception while handling client http traffic, closing connection [id: 0x80a4e5db, L:0.0.0.0/0.0.0.0:9200 ! R:/172.19.0.3:60480]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
        at sun.security.ssl.Alert.createSSLException(Alert.java:128) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:672) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:627) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:443) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:422) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
        ... 15 more

@jabaa
Copy link

jabaa commented Jun 28, 2019

Why is this issue closed? I have the same problem. Is there some solution for this? Can't finish the setup.

@BeunHaas34
Copy link

same issue here

@gnunesmoura
Copy link

I'm having the same issue.

@zentale zentale mentioned this issue Nov 26, 2019
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixing hanging issue - adding ulimits to elasticsearch zentale/stack-docker
9 participants
@fxdgear @rajiff @vanosg @Crysalist @BeunHaas34 @jabaa @ohing504 @gnunesmoura @minhtd1981