Recommendation for health checks / discovering unsealed vaults about vault-boshrelease HOT 13 CLOSED

@drnic We removed the script from Concourse due to the security reasons. We didn't want to keep unseal keys in one place and we decided to leave the unsealing part as the manual job. Now we have a Concourse job that checks if any Vault instance is sealed and if yes, then it sends an email notification to key holders.

from vault-boshrelease.

Just an aside, if you use the safe-boshrelease (https://github.com/cloudfoundry-community/safe-boshrelease), then safe seal and safe unseal start working and do the magical needful.

from vault-boshrelease.

Are you recommending adding additional health checks above and beyond what the Consul backend provides? Or are you talking about registering with an external Consul instance?

from vault-boshrelease.

Maybe I'm misunderstanding--what I'm trying to ask is how you make sure that clients only talk to unsealed vault instances. If clients look up vault instances via consul dns, then you're already done. Or if clients don't have access to consul, you might use an external load balancer with a health check.

from vault-boshrelease.

I don't think we can make sure that clients only talk to unsealed Vaults, not without explicit Vault-aware support in the client binary / library.

from vault-boshrelease.

An example of Vault-aware client machinery is the forthcoming HA support in safe, once we finish reviewing and merge this PR: Qarik-Group/safe#45

from vault-boshrelease.

Vault itself doesn't know how to prevent clients from accessing sealed instances, but it does include a health check endpoint that can be used by a load balancer to remove sealed instances from service--that's exactly what the example load balancer in the vault docs at https://github.com/hashicorp/vault/blob/master/terraform/aws/main.tf#L76-L107 is used for. And like I mentioned earlier, if clients use consul dns to find vault instances, the vault.service.consul name will resolve to all unsealed vault instances. My question is which of those approaches you use to prevent clients from talking to sealed instances, if any. It sounds like the answer is none of the above, but that you're going to solve the issue using your client instead.

In any case, I think it would be useful to include a note on this in the docs, even if only to say that you recommend using safe, which handles the issue, and that users of different clients will want to solve the problem of sealed instances on their own.

from vault-boshrelease.

I bypassed this issue on AWS by setting a health check on HTTPS:8200/v1/sys/leader which returns 500 only on sealed instances so they are treated by ELB as out of service.

from vault-boshrelease.

We've done similar things with haproxy fronting Vault, but have run into minor issues during the post-update "everything-is-sealed" scenario. How are you solving that in AWS, @simonkey007 ?

from vault-boshrelease.

We have a script that every minute checks all the instances, one by one and unseal them if they are sealed.

from vault-boshrelease.

I wonder if it would be worthwhile to package that up as a post-start or post-deploy script (optional, controlled by vault.auto_unseal: true in the manifest) or not.

from vault-boshrelease.

@jmcarp @simonkey007 just checking to see if you've looked any further into this; Thanks :)

from vault-boshrelease.

@simonkey007 is your (cron?) script still the solution that works for you?

from vault-boshrelease.