Don't Use PW in ENV vars and default SSL on #39

fxdgear · 2018-05-08T16:18:25Z

New Description to account for changes in direction during the life of this PR and feedback received.

Don't use ENV vars to specify passwords for running containers
Setup script and yml file will

generate an ELASTIC_PASSWORD if one is not provided by the user at script execution time.
generate certs for security
generate keystores to store passwords securely

Use docker secrets (aka a volume mount in compose) to securely include the sensitive info (keystores, certs, etc) in containers at container run time.
Each configuration file is now included in the config directory and no longer are configs managed via env vars.

this makes it easier to see what the configs are
make changes to them for personal use or experimentation
configs pull sensitive information from keystores previously generated.

Security is enabled by default and communication between services is done over ssl/tls.

with the exception of kibana. Kibana runs on port 5601 without ssl enabled. This is to avoid the hoops of navigating to a website with an untrusted (self signed) cert. But communication between Kibana and Elasticsearch are secured and use SSL.

xeraa · 2018-05-08T16:26:11Z

config/apm-server/apm-server.yml

+  hosts: ['elasticsearch:9200']
+  protocol: "https"
+  username: elastic
+  # Read PW from auditbeat.keystore


Nitpick: apm.keystore

xeraa · 2018-05-08T16:27:37Z

config/filebeat/filebeat.yml

+  hosts: ['elasticsearch:9200']
+  protocol: "https"
+  username: elastic
+  # Read PW from auditbeat.keystore


Ok, this must be a copy paste issue :)

hah yeah it is I'll fix.

xeraa · 2018-05-08T16:44:50Z

config/filebeat/filebeat.yml

+    reload.enabled: false
+
+processors:
+- add_cloud_metadata:


Shouldn't this be add_docker_metadata? Here and in the 3 other Beats configs

Oh? Quite possibly. I just copied out the config from each container as it ran in the master branch.

I wasn’t checking for correctness.

But I suppose this exercise is a good one given that we can now analyze more easily the config for each service.

Ah, I didn't check that. I've created #40 to keep track of this — no point in adding that to this PR as well.

elasticdog

This all makes sense to me and looks much cleaner! Loving the direction.

Beyond @jarpy's make on Windows concern, my only real comment is that checking in static public/private keys (even as an example) is a bit of a fishy smell to me. I could see where someone might accidentally run something production-y using those...could we instead generate them as part of the initial setup step?

fxdgear · 2018-05-09T02:04:41Z

@elasticdog Make is for Linux/Mac people. Windows users can use the multiple -f in docker compose.

I’ll upstate to dynamically create keystores and certs.

Thanks for the feedback.

jarpy · 2018-05-09T05:02:24Z

There's something nasty happening with file permissions. When I bring up the stack I see this:

heartbeat            | Exiting: error loading config file: config file ("heartbeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/heartbeat/heartbeat.yml')
heartbeat exited with code 1
metricbeat           | Exiting: error loading config file: config file ("metricbeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/metricbeat/metricbeat.yml')
metricbeat exited with code 1
packetbeat           | Exiting: error loading config file: config file ("packetbeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/packetbeat/packetbeat.yml')
auditbeat            | Exiting: error loading config file: config file ("auditbeat.yml") must be owned by the beat user (uid=0) or root
packetbeat exited with code 1
filebeat             | Exiting: error loading config file: config file ("filebeat.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/filebeat/filebeat.yml')
filebeat exited with code 1
auditbeat exited with code 1
apm_server           | Exiting: error loading config file: config file ("apm-server.yml") can only be writable by the owner but the permissions are "-rw-rw-r--" (to fix the permissions use: 'chmod go-w /usr/share/apm-server/apm-server.yml')

Same behaviour on Windows and Linux.

jarpy · 2018-05-09T05:10:06Z

I'm in two minds about running Kibana on HTTPS. It means that the first thing a brand new user sees is an error page with the word "unsafe" in it. We know that's just a self-signed cert, but it doesn't give that "it just works" feeling.

fxdgear · 2018-05-09T17:42:22Z

@jarpy I'm happy you ran into the permissions issue. I kinda thought it might happen. But wasn't sure. I'll update. thank you

* upgrade docker-compose file version * upgrade elasticstack TAG * use secrets instead of PW in ENV vars * Make it easy to use secrets * Don't teach people to use passwords in ENV vars * just as easy to use a docker secret as it is to use a bind mounted volume * adding setup.yml to create 1. passwords 2. keystores 3. certs adding health checks update the readme turn off ssl for kibana endbpoint to avoid confusion for new users

…image -platinum

fxdgear · 2018-07-02T21:59:09Z

@elastic/infra if anyone would like to review this PR I'm happy to answer any questions. i think it's ready to go now for real world use.

tvernum · 2018-07-03T00:31:01Z

Keystores are pre-configured with the "changeme" password

Assuming I understand this correctly, can we please not do this.
We went to a lot of trouble to get rid of default passwords from ES, please don't add them back in via docker.

Ping: @elastic/es-security

fxdgear · 2018-07-03T15:53:58Z

@tvernum

What are you trying to understand. I'm creating a keystore and then adding the password "changeme" to it.

If you're wanting to get away from the password "changeme" I can default it to something else.

the other option is I can use an automatically generated password and output it in the setup script after everything is done. Something like:

The elastic password is: blahblahblah

Will that work?

EDIT: @tvernum also I realized after re-reading my PR description it needs to be changed to reflect the fact that the keystores (and any senstiive) info is no longer included like I had originally done in this PR.

Keystores and certs are now generated using the setup.sh script in the setup.yml docker-compose file. This allows for every single instance of this project when run to be using unique sensitive data.

tvernum · 2018-07-04T00:42:37Z

@fxdgear Sorry, I had forgotten that stack-docker has historically had a pre-configured password ( 😢 😭 🤢 😡 )

I would truly love if we never shipped anything that had any preconfigured password in it. Even for things that are intended to be examples rather than production ready - users take what we provide as a gold standard and do likewise.

If it's possible for the setup process to ask for a password, or generate a password that would be a wonderful step forwards.

Also updated to echo the pw for the user when the script finishes.

fxdgear · 2018-07-05T19:51:32Z

@tvernum Does the latest commit work for you?
e1306b4

thanks for taking the time to review this!

Crazybus

I tried to run stack-docker blindly following the projects readme (as I would expect our users to do). I ran into a couple of issues which can be quickly fixed up.

Crazybus · 2018-07-06T05:43:42Z

README.md


 Point a browser at [`http://localhost:5601`](http://localhost:5601) to see the results.
+> *NOTE*: Elasticsearch is now setup with self-signed certs.
+
+Log in with `elastic` / `changeme`.


This should be updated to explain that the autogenerated password needs to be used if you don't set it yourself.

+1 thanks will do.

Crazybus · 2018-07-06T05:46:33Z

Makefile

+		centos:7  "/usr/local/bin/setup-permissions.sh"
+
+setup_users:
+	docker-compose run --no-deps -e ELASTIC_PASSWORD=changeme elasticsearch /usr/local/bin/setup-users.sh


This hardcoded password doesn't work anymore here. This will need to use the autogenerted password that you have now added.

Crazybus · 2018-07-06T05:48:32Z

Makefile

+	docker-compose run --no-deps -e ELASTIC_PASSWORD=changeme heartbeat /usr/local/bin/setup-beat.sh heartbeat
+
+setup_metricbeat:
+	docker-compose run --no-deps -e ELASTIC_PASSWORD=changeme metricbeat /usr/local/bin/setup-beat.sh metricbeat


I didn't dive into it but when I try to run make setup_metricbeat with the ELASTIC_PASSWORD updated to my autogenerated one I'm getting Read-only file system errors.

chmod: changing permissions of '/usr/share/metricbeat/metricbeat.yml': Read-only file system

fxdgear · 2018-07-06T15:24:13Z

@Crazybus 💥 thanks for the feedback. I forgot that I modified the Makefile. I'm gonna set that back to it's normal thing cause I put all the setup in a docker container. There's no need to use the make file anymore as a general user.

update docs on readme for how to use and get the elastic password

fxdgear · 2018-07-06T15:55:29Z

@jarpy @tvernum @xeraa @Crazybus

You all have commented on this PR at some point. I'd like to invite you to review it again.

I feel like I've addressed everyones comments and concerns, and would like to get this "done" with so it's not something I have lingering anymore :)

Thanks for all your assistance with this.

jarpy · 2018-07-09T02:45:30Z

I think this breaks on Windows because it depends on the Docker Unix socket.

λ  docker-compose -f setup.yml up
WARNING: The PWD variable is not set. Defaulting to a blank string.
Creating stack-docker_setup_1 ... error

ERROR: for stack-docker_setup_1  Cannot create container for service setup: b'Mount denied:\nThe source path "\\\\var\\\\run\\\\docker.sock:/var/run/docker.sock"\nis not a valid Windows path'

ERROR: for setup  Cannot create container for service setup: b'Mount denied:\nThe source path "\\\\var\\\\run\\\\docker.sock:/var/run/docker.sock"\nis not a valid Windows path'
ERROR: Encountered errors while bringing up the project.

fxdgear · 2018-07-09T07:40:34Z

@jarpy ahh thanks for this. you have to specify some env vars for a windows user and I forgot to put those in the read me. I'll add thanks. :)

Crazybus · 2018-07-09T10:02:49Z

Makefile

@@ -28,3 +28,4 @@ $(TARGETS:%=%-checkout):

 $(TARGETS:%=%-clean):
 	rm -rf stack/$(@:%-clean=%)


Can we make sure make clean also removes the keystores? Took me a while to figure out why this was causing me problems when trying to spin up a fresh cluster after doing a make clean and a delete all my docker data with fire

* remove keystores * do proper docker-compose down * remove volumes and networks make it fresh

fxdgear · 2018-07-09T16:26:05Z

@Crazybus I made the makefile a bit more exhaustive wrt to the clean target. PTAL :)

thank you for taking the time to review!

fxdgear · 2018-07-09T16:26:55Z

@jarpy PTAL I added a new commit which updates the README to account for the 2 env vars needed to make this work on Windows: 6d1941f

jarpy · 2018-07-10T07:24:19Z

Thanks, @fxdgear.

Still not behaving itself on my Windows machine, I'm afraid.

λ  (pwd).path
C:\Users\jarpy\src\stack-docker

λ  $Env:COMPOSE_CONVERT_WINDOWS_PATHS=1

λ  $Env:PWD="/c/Users/jarpy/src/stack-docker"

λ  docker-compose -f setup.yml up
Starting stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | cat: can't open './scripts/setup.sh': No such file or directory
stack-docker_setup_1 exited with code 0

joshbressers · 2018-07-11T17:40:45Z

scripts/setup.sh

@@ -4,6 +4,9 @@ chown 1000 -R "$confdir"
 find "$confdir" -type f -name "*.keystore" -exec chmod go-wrx {} \;
 find "$confdir" -type f -name "*.yml" -exec chmod go-wrx {} \;

+PW=$(date +%s | sha256sum | base64 | head -c 16 ;)


We shouldn't do this to generate the password, it will be predictable. Better to grab some data from /dev/urandom or use openssl

jordansissel · 2018-07-11T22:20:58Z

Possible error conditions I've found:

On second attempts to -f setup.yml up, some files may already exist:

When elasticsearch.keystore exists, it is not regenerated. Still, a new ELASTIC_PASSWORD is generated and used in Kibana and Logstash, but not set in elasticsearch.keystore. Workaround: Remove elasticsearch.keystore and -f setup up again.
Kibana's SSL files, when they already exist, may not be repopulated. Workaround: Remove all new/modified files in config/ and git checkout . to restore normal files, then -f setup up again.

fxdgear · 2018-07-11T22:33:05Z

@jordansissel if you have make installed you can run a make clean which will solve this as well. BUT unfortuanly make isn't installed by default on windows :(

Thank you so much for all your help on windows testing!

russcam · 2018-07-11T22:39:50Z

README.md

+  * `PWD=/path/to/checkout/for/stack-docker`
+    * for example I use the path: `/c/Users/nick/elastic/stack-docker`
+  * You can set these two ways:
+    1. in powershell use: `$Env:COMPOSE_CONVERT_WINDOWS_PATHS=1`


This will only set the variable for the current PowerShell session, which may be OK for this?

The equivalent of option 2. would be

[Environment]::SetEnvironmentVariable("COMPOSE_CONVERT_WINDOWS_PATHS", "1", "Machine")

As per number 2, you'd need to start a new session after this, for the change to be picked up (or refresh the environment from the registry).

@russcam thanks! I had no idea you could do that! ;)

Possible fixes: Use curl's `-u` flag for user:pass, or use hex instead of base64 for `openssl rand`. I chose `curl -u` The bug is that base64 includes `/`, so the `openssl rand -base64 16` password could contain a `/` and make curl think that the username is really the hostname: ``` curl 'https://elastic:nfAvKVigT7Bd7R60/o+1OQ==@elasticsearch:9200/' curl: (6) Could not resolve host: elastic; Unknown error ``` The bug manifests as setup_logstash and setup_kibana getting stuck waiting for Elasticsearch to be online, but never succeeds because the curl invocation is wonky.

jordansissel · 2018-07-12T01:41:03Z

Tested latest commit, and it works for me (Tested on Windows 10 / powershell).

fxdgear · 2018-07-12T17:23:29Z

@Crazybus can you please review again. I hope I've addressed all your changes.
@jordansissel +1 thanks for the patch super helpful.
@russcam Can you please review the README again. i've taken your suggestion and added it.
@jarpy When you have time lets sit down and see where you are blocked.

Crazybus

LGTM! All is working well now when I blindly follow the readme 👍

As we discussed filebeat currently doesn't have any incoming data. An idea would be to mount the docker socket into the container and show the docker logs inside of this demo. However this is out of scope for this PR.

russcam · 2018-07-16T00:21:15Z

The PowerShell environment variable change looks good @fxdgear 👍

fxdgear changed the title ~~[WIP] Don't Use PW in ENV vars~~ [WIP] Don't Use PW in ENV vars and default SSL on May 8, 2018

xeraa reviewed May 8, 2018

View reviewed changes

elasticdog reviewed May 9, 2018

View reviewed changes

xeraa mentioned this pull request May 9, 2018

Add add_docker_metadata to the Beats configurations #40

Open

fxdgear force-pushed the nick/dont_use_pw_in_env branch 2 times, most recently from d0fb0d2 to 1a1f656 Compare June 22, 2018 18:54

fxdgear force-pushed the nick/dont_use_pw_in_env branch from eb44d57 to 87ef7dc Compare June 22, 2018 19:02

fxdgear added 3 commits June 22, 2018 13:10

.env password is needed to seed the whole process, remove old docker …

de3852b

…image -platinum

remove https from kibana, makes stack a bit easier to get started

8494758

remove zip file and allow setp.yml to work with windows

84b501e

fxdgear changed the title ~~[WIP] Don't Use PW in ENV vars and default SSL on~~ Don't Use PW in ENV vars and default SSL on Jul 2, 2018

if env ELASTIC_PASSWORD not provided by user generate a pw.

e1306b4

Also updated to echo the pw for the user when the script finishes.

Crazybus suggested changes Jul 6, 2018

View reviewed changes

undo changes to make file as they aren't needed anymore

103f439

update docs on readme for how to use and get the elastic password

Crazybus suggested changes Jul 9, 2018

View reviewed changes

fxdgear added 2 commits July 9, 2018 10:08

Make the makefile clean up more things.

4b5e045

* remove keystores * do proper docker-compose down * remove volumes and networks make it fresh

updated readme with more windows instructions re env vars

6d1941f

fxdgear mentioned this pull request Jul 10, 2018

Ability to Run Logstash with Centralized Pipeline Management #16

Open

joshbressers reviewed Jul 11, 2018

View reviewed changes

use openssl to generate password

ecb66ef

russcam reviewed Jul 11, 2018

View reviewed changes

fxdgear added 2 commits July 12, 2018 11:19

force recreation of es keystore when running setup

b204526

update readme with ways to set env vars in powershell

442f7e0

more notes for windows users regarding path structure

5d4cd32

Crazybus approved these changes Jul 13, 2018

View reviewed changes

fxdgear merged commit 898d08b into master Jul 13, 2018

		@@ -28,3 +28,4 @@ $(TARGETS:%=%-checkout):

		$(TARGETS:%=%-clean):
		rm -rf stack/$(@:%-clean=%)

Don't Use PW in ENV vars and default SSL on #39

Don't Use PW in ENV vars and default SSL on #39

Conversation

fxdgear commented May 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticdog left a comment

Choose a reason for hiding this comment

fxdgear commented May 9, 2018

jarpy commented May 9, 2018

jarpy commented May 9, 2018

fxdgear commented May 9, 2018

fxdgear commented Jul 2, 2018

tvernum commented Jul 3, 2018

fxdgear commented Jul 3, 2018 • edited

tvernum commented Jul 4, 2018

fxdgear commented Jul 5, 2018

Crazybus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fxdgear commented Jul 6, 2018 • edited

fxdgear commented Jul 6, 2018

jarpy commented Jul 9, 2018 • edited

fxdgear commented Jul 9, 2018

Choose a reason for hiding this comment

fxdgear commented Jul 9, 2018

fxdgear commented Jul 9, 2018

jarpy commented Jul 10, 2018

Choose a reason for hiding this comment

jordansissel commented Jul 11, 2018

fxdgear commented Jul 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jordansissel commented Jul 12, 2018

fxdgear commented Jul 12, 2018

Crazybus left a comment

Choose a reason for hiding this comment

russcam commented Jul 16, 2018

fxdgear commented May 8, 2018 •

edited

fxdgear commented Jul 3, 2018 •

edited

fxdgear commented Jul 6, 2018 •

edited

jarpy commented Jul 9, 2018 •

edited