Skip to content
This repository has been archived by the owner on Apr 12, 2022. It is now read-only.

invalid mount config for type "bind": bind source path does not exist #65

Open
perfecto25 opened this issue Nov 26, 2018 · 17 comments
Open

Comments

@perfecto25
Copy link

perfecto25 commented Nov 26, 2018

Hello, having some issues bringing up the stack,

root@min1 /e/elastic# uname -a
Linux min1 3.10.0-862.14.4.el7.x86_64 elastic/stack-docker#1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux


root@min1 /e/elastic# docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

I ran the docker-compose -f setup.yaml up, the entire setup is done and gives me a password.


root@min1 /e/elastic# docker-compose -f setup.yml up --remove-orphans
Creating network "elastic_default" with the default driver
Creating elastic_setup_1_2ca56a7c094b ... done
Attaching to elastic_setup_1_2564d05a80d6
setup_1_2564d05a80d6 | Creating network "elastic_stack" with the default driver
setup_1_2564d05a80d6 | Found orphan containers (elastic_setup_1_2564d05a80d6) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating setup_elasticsearch ... done
Attaching to setup_elasticsearch
setup_1_2564d05a80d6 | setup_elasticsearch    | Determining if x-pack is installed...
setup_1_2564d05a80d6 | setup_elasticsearch    | === CREATE Keystore ===
setup_1_2564d05a80d6 | setup_elasticsearch    | Elastic password is: "zqEag8W0Q5unC56UkEmnkg=="
setup_1_2564d05a80d6 | setup_elasticsearch    | Created elasticsearch keystore in /usr/share/elasticsearch/config
setup_1_2564d05a80d6 | setup_elasticsearch    | Setting bootstrap.password...
setup_1_2564d05a80d6 | setup_elasticsearch    | === CREATE SSL CERTS ===
setup_1_2564d05a80d6 | setup_elasticsearch    | Remove old ca zip...
setup_1_2564d05a80d6 | setup_elasticsearch    | Creating docker-cluster-ca.zip...
setup_1_2564d05a80d6 | setup_elasticsearch    | CA directory exists, removing...
setup_1_2564d05a80d6 | setup_elasticsearch    | Unzip ca files...
setup_1_2564d05a80d6 | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster-ca.zip
setup_1_2564d05a80d6 | setup_elasticsearch    |    creating: /config/ssl/ca/
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.crt   
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.key   
setup_1_2564d05a80d6 | setup_elasticsearch    | Remove old docker-cluster.zip zip...
setup_1_2564d05a80d6 | setup_elasticsearch    | Create cluster certs zipfile...
setup_1_2564d05a80d6 | setup_elasticsearch    | Unzipping cluster certs zipfile...
setup_1_2564d05a80d6 | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster.zip
setup_1_2564d05a80d6 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt  
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key  
setup_1_2564d05a80d6 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt  
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key  
setup_1_2564d05a80d6 | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt  
setup_1_2564d05a80d6 | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key  
setup_1_2564d05a80d6 | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1_2564d05a80d6 | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1_2564d05a80d6 | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1_2564d05a80d6 | setup_elasticsearch exited with code 0
setup_1_2564d05a80d6 | Found orphan containers (elastic_setup_1_2564d05a80d6) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating elasticsearch ... done
Creating setup_kibana   ... done
Creating setup_logstash ... done
Attaching to setup_kibana, setup_logstash
setup_1_2564d05a80d6 | setup_kibana           | -rw-rw-r--. 1 root root 1200 Nov 26 16:39 /usr/share/kibana/config/ca/ca.crt
setup_1_2564d05a80d6 | setup_logstash         | -rw-rw-r--. 1 root root 1200 Nov 26 16:39 /usr/share/logstash/config/ca/ca.crt
setup_1_2564d05a80d6 | setup_logstash         | {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: passwords must be at least [6] characters long;"}],"type":"validation_exception","reason":"Validation Failed: 1: passwords must be at least [6] characters long;"},"status":400}=== CREATE Keystore ===
setup_1_2564d05a80d6 | setup_logstash         | Remove old logstash.keystore
setup_1_2564d05a80d6 | setup_kibana           | {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: passwords must be at least [6] characters long;"}],"type":"validation_exception","reason":"Validation Failed: 1: passwords must be at least [6] characters long;"},"status":400}=== CREATE Keystore ===
setup_1_2564d05a80d6 | setup_kibana           | Remove old kibana.keystore
setup_1_2564d05a80d6 | setup_kibana           | Created Kibana keystore in /usr/share/kibana/data/kibana.keystore
setup_1_2564d05a80d6 | setup_kibana           | Setting elasticsearch.password: "zqEag8W0Q5unC56UkEmnkg=="
setup_1_2564d05a80d6 | setup_kibana exited with code 0
setup_1_2564d05a80d6 | setup_logstash         | 
setup_1_2564d05a80d6 | setup_logstash         | WARNING: The keystore password is not set. Please set the environment variable `LOGSTASH_KEYSTORE_PASS`. Failure to do so will result in reduced security. Continue without password protection on the keystore? [y/N] Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
setup_1_2564d05a80d6 | setup_logstash         | Setting ELASTIC_PASSWORD...
setup_1_2564d05a80d6 | setup_logstash         | 
setup_1_2564d05a80d6 | setup_logstash         | Enter value for ELASTIC_PASSWORD: Added 'elastic_password' to the Logstash keystore.
setup_1_2564d05a80d6 | setup_logstash exited with code 0
setup_1_2564d05a80d6 | Found orphan containers (elastic_setup_1_2564d05a80d6) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
setup_1_2564d05a80d6 | elasticsearch is up-to-date
Creating kibana ... done
Creating setup_packetbeat ... 
setup_1_2564d05a80d6 | Creating setup_metricbeat ... 
setup_1_2564d05a80d6 | Creating setup_filebeat   ... 
Creating setup_packetbeat ... done
Creating setup_metricbeat ... done
Creating setup_filebeat   ... done
setup_1_2564d05a80d6 | 
Creating setup_apm_server ... done
Creating setup_heartbeat  ... done
setup_1_2564d05a80d6 | 
setup_1_2564d05a80d6 | ERROR: for setup_auditbeat  Cannot start service setup_auditbeat: linux spec capabilities: Unknown capability to add: "CAP_AUDIT_READ"
setup_1_2564d05a80d6 | Encountered errors while bringing up the project.
setup_1_2564d05a80d6 | Setup completed successfully. To start the stack please run:
setup_1_2564d05a80d6 | 	 docker-compose up -d
setup_1_2564d05a80d6 | 
setup_1_2564d05a80d6 | If you wish to remove the setup containers please run:
setup_1_2564d05a80d6 | 	docker-compose -f docker-compose.yml -f docker-compose.setup.yml down --remove-orphans
setup_1_2564d05a80d6 | 
setup_1_2564d05a80d6 | You will have to re-start the stack after removing setup containers.
setup_1_2564d05a80d6 | 
setup_1_2564d05a80d6 | Your 'elastic' user password is: "zqEag8W0Q5unC56UkEmnkg=="
elastic_setup_1_2564d05a80d6 exited with code 0

when I startup the entire thing, getting bind errors,

root@min1 /e/elastic# docker-compose up 
Recreating elasticsearch ... done
Recreating kibana        ... done
Creating logstash        ... done
Creating metricbeat      ... 
Creating apm_server      ... error
Creating filebeat        ... 
Creating metricbeat      ... error
Creating packetbeat      ... 

ERROR: for apm_server  Cannot create container for service apm-server: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/apm-server/apm-server.keystore
Creating heartbeat       ... 
Creating auditbeat       ... error
ERROR: for metricbeat  Cannot create container for service metricbeat: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/metricbeat/metricbeat.keystore

Creating packetbeat      ... error
Creating filebeat        ... done

ERROR: for packetbeat  Cannot create container for service packetbeat: invalid mount config for type "bind": bind source path does noCreating heartbeat       ... done

ERROR: for auditbeat  Cannot create container for service auditbeat: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/auditbeat/auditbeat.keystore

ERROR: for apm-server  Cannot create container for service apm-server: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/apm-server/apm-server.keystore

ERROR: for metricbeat  Cannot create container for service metricbeat: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/metricbeat/metricbeat.keystore

ERROR: for packetbeat  Cannot create container for service packetbeat: invalid mount config for type "bind": bind source path does not exist: /etc/elastic/config/packetbeat/packetbeat.keystore
ERROR: Encountered errors while bringing up the project.

@simplycycling
Copy link

@perfecto25 I'm not sure what leads to this condition (I suspect it has something to do with the way docker-compose issues the mount command behind the scenes), but I was able to get this going by simply creating the directories, and then running docker-compose up -d.

@fxdgear
Copy link
Contributor

fxdgear commented Nov 28, 2018

@perfecto25 I'm curious if there's some permission errors going on and the process isn't able to create those keystores at the defined locations.

@melloware
Copy link

I just got bit by the same issue. creating the auditbeat.keystore solved my problem. Thanks for posting that solution.

@fxdgear
Copy link
Contributor

fxdgear commented Nov 28, 2018

I'm curious if the auditbeat service failing to start cause auditd isn't enabled for the kernel is breaking something in the whole process...

@qakart
Copy link

qakart commented Dec 13, 2018

I had the same issue and @fxdgear your solutions worked

@oori
Copy link

oori commented Jan 8, 2019

Same problem here, can't start the auditbeat container.
uname -r --> 4.15.0-43-generic

sudo apt-get install auditd audispd-plugins
sudo systemctl status auditd --> active (running)

docker-compose up
--> ERROR: for auditbeat Cannot start service auditbeat: invalid mount config for type "bind": bind source path does not exist: /home/user/stack-docker/config/auditbeat/auditbeat.keystore

For the test, if I touch /home/user/stack-docker/config/auditbeat/auditbeat.keystore and docker-compose up auditbeat, the container manages to start, but I get: Exiting: could not initialize the keystore: keystore format doesn't match expected version: 'v1' got '', hence - it can read the file.. (but ofcourse, it's invalid, empty).

All other containers are up, no permission issue in the config folder..
any idea?

@fxdgear
Copy link
Contributor

fxdgear commented Jan 8, 2019

Can you run auditbeat outside a container?

@oori
Copy link

oori commented Jan 8, 2019

I guess so... I've quickly did a test - installed auditbeat deb, and started the service. it's running. but haven't really tested thoroughly, @fxdgear please point me at specifics.

@fxdgear
Copy link
Contributor

fxdgear commented Jan 14, 2019

@oori soo I've found that auditbeat can be kinda troublesome on linux. I personally run arch and I don't have auditd enabled and as a result auditbeat fails to start. Since this project is designed to be for "demo purposes" I consider not seeing audit beat data on my linux laptop ok cause I know that auditbeat won't run.

I'd suggest maybe doing a clean up (containers, volumes, networks etc...) and trying again.

@mjpowersjr
Copy link

Hopefully the referenced pull request clears up some of these issues. The setup script seems to hang for me, I suspect it should be doing more work past the output below.

Environment:

  • Docker version 18.09.1, build 4c52b90
  • Ubuntu 18.10
$ docker-compose -f setup.yml up
WARNING: Found orphan containers (setup_kibana, setup_logstash) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating setup_elasticsearch ... done
Attaching to setup_elasticsearch
setup_1  | setup_elasticsearch    | Determining if x-pack is installed...
setup_1  | setup_elasticsearch    | === CREATE Keystore ===
setup_1  | setup_elasticsearch    | Elastic password is: 4F7BBILOZavEQjQyLIN2Cg==
setup_1  | setup_elasticsearch    | Created elasticsearch keystore in /usr/share/elasticsearch/config
setup_1  | setup_elasticsearch    | Setting bootstrap.password...
setup_1  | setup_elasticsearch    | === CREATE SSL CERTS ===
setup_1  | setup_elasticsearch    | Remove old ca zip...
setup_1  | setup_elasticsearch    | Creating docker-cluster-ca.zip...
setup_1  | setup_elasticsearch    | CA directory exists, removing...
setup_1  | setup_elasticsearch    | Unzip ca files...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster-ca.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/ca/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.crt   
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.key   
setup_1  | setup_elasticsearch    | Remove old docker-cluster.zip zip...
setup_1  | setup_elasticsearch    | Create cluster certs zipfile...
setup_1  | setup_elasticsearch    | Unzipping cluster certs zipfile...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key  
setup_1  | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1  | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1  | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1  | setup_elasticsearch exited with code 0
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating elasticsearch ... done
Recreating setup_kibana   ... done
Recreating setup_logstash ... done
Attaching to setup_kibana, setup_logstash
setup_1  | setup_kibana           | -rw-rw-r-- 1 root root 1200 Jan 30 12:12 /usr/share/kibana/config/ca/ca.crt
setup_1  | setup_logstash         | -rw-rw-r-- 1 root root 1200 Jan 30 12:12 /usr/share/logstash/config/ca/ca.crt

@fxdgear
Copy link
Contributor

fxdgear commented Jan 30, 2019

@mjpowersjr I'm guessing that something failed to start in Elasticsearch. And what's happening is the setup for kibana and logstash are hung cause elasticsearch isn't up (or hasn't come up yet).

If you like you can take a look at the referenced PR and give that a shot. Please note that the README has changed as well, so please read over that too.

@azizur
Copy link

azizur commented Apr 30, 2019

Looks like this issue has not been resolved in master branch.

@techdragon157
Copy link

I'm getting the same issue as mjpowersjr

Managed to get it working at some point but its hit or miss. Any idea when this can be resolved?

@sahil311289
Copy link

Hopefully the referenced pull request clears up some of these issues. The setup script seems to hang for me, I suspect it should be doing more work past the output below.

Environment:

  • Docker version 18.09.1, build 4c52b90
  • Ubuntu 18.10
$ docker-compose -f setup.yml up
WARNING: Found orphan containers (setup_kibana, setup_logstash) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Starting stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating setup_elasticsearch ... done
Attaching to setup_elasticsearch
setup_1  | setup_elasticsearch    | Determining if x-pack is installed...
setup_1  | setup_elasticsearch    | === CREATE Keystore ===
setup_1  | setup_elasticsearch    | Elastic password is: 4F7BBILOZavEQjQyLIN2Cg==
setup_1  | setup_elasticsearch    | Created elasticsearch keystore in /usr/share/elasticsearch/config
setup_1  | setup_elasticsearch    | Setting bootstrap.password...
setup_1  | setup_elasticsearch    | === CREATE SSL CERTS ===
setup_1  | setup_elasticsearch    | Remove old ca zip...
setup_1  | setup_elasticsearch    | Creating docker-cluster-ca.zip...
setup_1  | setup_elasticsearch    | CA directory exists, removing...
setup_1  | setup_elasticsearch    | Unzip ca files...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster-ca.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/ca/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.crt   
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.key   
setup_1  | setup_elasticsearch    | Remove old docker-cluster.zip zip...
setup_1  | setup_elasticsearch    | Create cluster certs zipfile...
setup_1  | setup_elasticsearch    | Unzipping cluster certs zipfile...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key  
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt  
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key  
setup_1  | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1  | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1  | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1  | setup_elasticsearch exited with code 0
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating elasticsearch ... done
Recreating setup_kibana   ... done
Recreating setup_logstash ... done
Attaching to setup_kibana, setup_logstash
setup_1  | setup_kibana           | -rw-rw-r-- 1 root root 1200 Jan 30 12:12 /usr/share/kibana/config/ca/ca.crt
setup_1  | setup_logstash         | -rw-rw-r-- 1 root root 1200 Jan 30 12:12 /usr/share/logstash/config/ca/ca.crt

docker-compose -f setup.yml up --> This helped me setup everything. Thank you!

@jhow31
Copy link

jhow31 commented May 30, 2019

This is, Setup is dosent creating de .keystore in all diretorys.
I will try to copy then to others.

@jhow31
Copy link

jhow31 commented Jun 8, 2019

Dont work, just copy the keystore... This is a error inside the development... I choose to use a total source, without docker.
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-centos-7

@paul-vu
Copy link

paul-vu commented Sep 16, 2019

I had an issue with this today. My problem was with autocrlf. I had cloned down the repo with autocrlf=true, and that caused the compose to hang and not create any of the keystores because of the new line characters in the shell scripts. When I set "git config core.autocrlf false" and re-cloned, then it started working for me.

Edit: Forgot to mention, for the issue creator's original issue. The issue is with kernel 3.10 not knowing what "CAP_AUDIT_READ" is. Source: #14

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests