Optionally seperate S3 object directory by metadata about core-dump-handler HOT 2 CLOSED

Hey @isbee

Option 1 of multiple S3 buckets per vendor per cluster is something that can be done today.
Just configure different S3 locations per install of CDH.
Option 2 of a single S3 bucket with nested directories seems like it could become fairly cumbersome as S3 is a key/value store so directories would really be a path convention but I guess it would depend on your use case.

Maybe making it possible to provide mechanism that allows you to define the S3 path would be enough to cover the option 2 requirement?

On the namespace- that is currently provided in the json metadata files.
ps, runtime and pod json files provide this information in some cases. You will have to test your provider.
Unfortunately there is no way for the pod to pass this information to the composer at dump time and the operating system has a very limited set of parameters it can capture.

I do like the idea of capturing the node hostname and will provide that as an addition property in the dump-info.json

As a general comment you may wish to consider a post processor that takes the items from the S3 and stores them in a more structured manner for your use case.
I think search, personal data scrubbing, alerting are obvious requirements that are probably beyond the scope of this project but if there is a "simple" solution to address these then I am willing to consider accepting PRs but it should be discussed prior to development.

Let me know your thoughts on the dynamic path feature and the post processor

from core-dump-handler.

As there are no further comments I am going to close this but feel free to reopen if you have some more detail on the paths requirements

from core-dump-handler.