Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Next steps after terraform apply #95

Open
arvindamirtaa opened this issue May 5, 2021 · 8 comments · May be fixed by #99
Open

Next steps after terraform apply #95

arvindamirtaa opened this issue May 5, 2021 · 8 comments · May be fixed by #99

Comments

@arvindamirtaa
Copy link

arvindamirtaa commented May 5, 2021 •
edited

Question
I think I'm missing a step. I cloned the repo, set the vars, ran terraform init, terraform plan and then terraform apply.

I got Apply complete! Resources: 41 added, 0 changed, 0 destroyed..

Then, I ran the root_token_decrypt_command and got the token. Set VAULT_ADDR, VAULT_TOKEN & VAULT_CAPATH as pert he README.md.

Assuming this is all correct, I should be able to interact with vault. Except I get...

Error enabling: Post "https://<IP>/v1/sys/mounts/secret": dial tcp <IP>:443: connect: connection refused

Upon digging a little further, the load balancer is reporting that the health check is failing for all 3 instances.

Considerations
Is there a step I'm missing? Any pointers on what I can look into next?
@agates4
Copy link

agates4 commented Aug 10, 2021

this is happening for me too:
the load balancer is reporting that the health check is failing for all 3 instances.

@sethvargo looks like something happened where this terraform config no longer works out of the box - needs an update, this issue is reproducible

@agates4
Copy link

agates4 commented Aug 10, 2021 •
edited 

vault secrets enable -path=secret -version=2 kv
Error enabling: Post "https://<IP>/v1/sys/mounts/secret": EOF

I actually get this error rather than a connection refused error

@agates4
Copy link

agates4 commented Aug 10, 2021

image
all the pods are healthy upon Kubernetes inspection, it is just a failure of the health check (and also a failure of accessing the UI through port 8200)

@agates4
Copy link

agates4 commented Aug 10, 2021

logs for vault-0 vault-init:

2021/08/10 23:13:17 Starting the vault-init service...
2021/08/10 23:13:17 Head http://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused
2021/08/10 23:13:27 Vault is not initialized.
2021/08/10 23:13:27 Initializing...
2021/08/10 23:13:30 Encrypting unseal keys and the root token...
2021/08/10 23:13:30 Unseal keys written to gs://<censored>-vault-storage/unseal-keys.json.enc
2021/08/10 23:13:30 Root token written to gs://<censored>-vault-storage/root-token.enc
2021/08/10 23:13:30 Initialization complete.
2021/08/10 23:13:30 Next check in 10s
2021/08/10 23:13:40 Vault is initialized and unsealed.
2021/08/10 23:13:40 Next check in 10s
2021/08/10 23:13:51 Vault is initialized and unsealed.
2021/08/10 23:13:51 Next check in 10s
2021/08/10 23:14:01 Vault is initialized and unsealed.
2021/08/10 23:14:01 Next check in 10s
2021/08/10 23:14:11 Vault is initialized and unsealed.

logs for vault-0 vault

==> Vault server configuration:

      GCP KMS Crypto Key: vault-init
        GCP KMS Key Ring:  <censored>
         GCP KMS Project: <censored>
          GCP KMS Region:  <censored>
               Seal Type: gcpckms
             Api Address:  <censored>
                     Cgo: disabled
         Cluster Address: https://10.0.92.5:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
              Listener 2: tcp (addr: "10.0.92.5:8200", cluster address: "10.0.92.5:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: warn
                   Mlock: supported: true, enabled: true
                 Storage: gcs (HA available)
                 Version: Vault v1.2.1

==> Vault server started! Log data will stream in below:

2021-08-10T23:13:24.383Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2021-08-10T23:13:27.389Z [WARN]  core: stored keys supported on init, forcing shares/threshold to 1
2021-08-10T23:13:30.637Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed

@agates4
Copy link

agates4 commented Aug 11, 2021

this project is no longer supported ? @sethvargo

@agates4
Copy link

agates4 commented Aug 11, 2021

on further inspection:

I reverted this PR: #94
and then everything started working as normal.

cheers!

@agates4 agates4 mentioned this issue Aug 11, 2021
Merged
@bchrobot
Copy link
Contributor

bchrobot commented Aug 12, 2021 •
edited

PR #94 adds support for Kubernetes Service Registration. This project is still using Vault 1.2.1 (now 2 years old) by default, however, and Kubernetes Service Registration wasn't added until Vault 1.4. Vault is now at v1.8.1. Try updating the Vault version to 1.8.1 on sethvargo/vault-on-gke#master.

From the logs you posted, @agates4, it seems like there may be a separate Vault error unrelated to the updating of Pod labels

2021-08-10T23:13:24.383Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2021-08-10T23:13:27.389Z [WARN]  core: stored keys supported on init, forcing shares/threshold to 1
2021-08-10T23:13:30.637Z [WARN]  core: attempted unseal with stored keys, but vault is already unsealed

@agates4
Copy link

agates4 commented Aug 12, 2021

thank you @bchrobot !

using these values

variable "vault_container" {
  type        = string
  default     = "vault:1.8.1"
  description = "Name of the Vault container image to deploy. This can be specified like \"container:version\" or as a full container URL."
}

variable "vault_init_container" {
  type        = string
  default     = "sethvargo/vault-init:0.2.0"
  description = "Name of the Vault init container image to deploy. This can be specified like \"container:version\" or as a full container URL."
}

within variables.tf, the up-to-date code works as expected. Cheers 🥂

bchrobot added a commit to politics-rewired/vault-on-gke that referenced this issue Aug 12, 2021
@bchrobot
fix: update vault and vault-init versions
29fe647 
PR sethvargo#94 relies on a feature introduced in Vault 1.4.0. This updates Vault to the latest version
fixing sethvargo#95.
@bchrobot bchrobot linked a pull request Aug 12, 2021 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: update vault and vault-init versions politics-rewired/vault-on-gke
3 participants
@arvindamirtaa @bchrobot @agates4