s3 setup for archival storage about deployment-automation HOT 6 CLOSED

Yes, this is not merged yet. The options are:

  archival_storage_enabled: true
  archival_storage_s3_access_key: <...>
  archival_storage_s3_secret_key: <...>
  archival_storage_s3_region: eu-north-1
  archival_storage_s3_bucket: redpandabackup2
  archival_storage_upload_interval: 10000
  archival_storage_max_connections: 5

developer mode should be enabled, you also have to specify AWS security credentials, bucket should be created in advance, versioning and encryptions are not needed, upload_interval is in milliseconds (5 or 10 seconds is OK), max_connections sets number of simultaneous S3 uploads per-shard, values in 1-10 range should be fine

from deployment-automation.

thanks @Lazin! that info should be good to get me started (I think)

from deployment-automation.

@ivotron so we have three modes

1. ductape local
2. ducktape cluster
3. punisher

We're handling (1) with an extra docker container running a local s3 endpoint. For (2) and (3) I assume these share a common method for deployment?

from deployment-automation.

I renamed archival_storage_upload_interval to archival_storage_reconciliation_interval_ms. It has a sensible default so there is no need to specify it explicitly anywhere. Will create PR with doc tomorrow.

from deployment-automation.

Is this ticket tracking adding shadow indexing feature, or is this for some other (possibly more specific) issue? I didn't want to create an additional ticket if it wasn't needed.

from deployment-automation.

Created a new ticket for SI on AWS: #87

Associated PR: #86

from deployment-automation.