Why use S3 for storage instead of Consul? about terraform-aws-vault HOT 8 CLOSED

Read about them here:

https://www.vaultproject.io/docs/configuration/storage/consul.html
https://www.vaultproject.io/docs/configuration/storage/s3.html

from terraform-aws-vault.

For durability. S3 helps ensure that you don't lose all your secrets by accidentally killing off the Consul cluster. https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster#s3-bucket

from terraform-aws-vault.

Howdy. I see that we've now actually removed S3 backend in current code. I liked the durability provided by an S3 backend as @brikis98 mentioned above. I had already pinned my modules to 0.0.8, so I can keep the old model for a while before a transition. Is there a form of Consul deployment that can closely mimic the durability that an S3 backend provides? I have tested many times this feature, where I can tear down my whole environment, except for the S3 buckets, and bring it back up again, and not miss any secrets! :D Can I do that with Consul, as constructed with this module and examples? Thanks.

from terraform-aws-vault.

We did remove it, as HashiCorp recommends using Consul as the primary store. You could use S3 with this module by overriding the configuration. Alternatively, if someone wants to make a PR to the run-vault script that supports S3 as a (non-default) alternative to Consul, I'd also welcome that.

from terraform-aws-vault.

Consul is now the default backend, but with #41, you can now optionally enable S3 as a storage backend and use Consul solely for HA.

from terraform-aws-vault.

Hello everyone
I build a project and i want use a remote backend for the team can work, i saw that consul and s3 are a good options for that but i don't sure which is the best.

from terraform-aws-vault.

@brikis98 you mentioned that Hashicorp recommends consul as a primary backend but I can't find an explanation in the documentation of why this is the better idea.

The way I understand it these are the two options:

Option a) Use consul as only backend HA storage and do periodic snapshots to be copied over to a bucket, let's say every 15 minutes? Downside of this approach is that if I accidentally destroy a consul server I may lose up to 15 minutes data. Upgrading consul servers in an automated and disposable way can be a pain when following this approach and downtime may be unavoidable.

Option b) Use s3 as backend but also use consul as HA for locking. Here is where I get really confused. What happens if I grab a snapshot from option a and restore in option b? Would that overwrite the data on the bucket? Can you still do snapshots of consul when you are doing this? Or do you backup in different ways?

This is the kind of information that's missing in the documentation (Or I can't find it, if you know where please point me in the right direction). It would be helpful and would save time in experimenting and trying to figure things out on your own.

Another thing that seems very tedious is upgrading consul, the procedure described here:

https://www.consul.io/docs/upgrading.html

Feels very "retro".

I managed to automate the upgrading of vault itself with the method described here:

https://groups.google.com/forum/#!msg/terraform-tool/7Gdhv1OAc80/iNQ93riiLwAJ

Which works amazing, but doing that for consul sadly wouldn't work so well, so I am wondering if option b would be easier to automate an upgrade with no downtime.

from terraform-aws-vault.

https://learn.hashicorp.com/vault/operations/ops-reference-architecture is probably a good starting point.

from terraform-aws-vault.