Thanos Sidecar - Flush Endpoint about thanos HOT 9 OPEN

As someone having to orchestrate "flushing" outside of prometheus-operator currently, I would definitely leverage a /flush endpoint in thanos. I'd also much prefer it as an in-built command than a separate container like config-reloader.

It'd also be much easier to benefit from incremental progress on this, as I'd be able to remove a lot of my custom glue/wiring in my current setup, instead of having to wait for the changes to propagate all the way to prometheus-operator.

from thanos.

cc @ArthurSens @simonpasquier

from thanos.

I've added the ability to upload blocks to the blob storage using Thanos Tools, we can leverage this on Prometheus Operator by doing some hacks such as a preStop hook to flush tsdb and upload blocks. As we can see we already have feature requests from users implementing this on their own, as you can see prometheus-operator/prometheus-operator#6540.

If Thanos sidecar implements this API, how would you call it from Prometheus operator? Would you still use the preStop hook to call the API or at downscaling event time?

from thanos.

I've added the ability to upload blocks to the blob storage using Thanos Tools, we can leverage this on Prometheus Operator by doing some hacks such as a preStop hook to flush tsdb and upload blocks. As we can see we already have feature requests from users implementing this on their own, as you can see prometheus-operator/prometheus-operator#6540.

If Thanos sidecar implements this API, how would you call it from Prometheus operator? Would you still use the preStop hook to call the API or at downscaling event time?

The idea is invoke this on the scaling down event.

from thanos.

I am not against adding this functionality into Thanos sidecar.

By having this new endpoint I envision Prometheus Operator calling it before scaling down the instance, and after this endpoint returns success we can just delete the Statefulset and its Persistent volume.

Prometheus operator has its own sidecar like the config-reloader. Why this feature cannot be implemented in another binary deployed as a sidecar as well? The code can be maintained in the Prometheus operator project.

from thanos.

I am not against adding this functionality into Thanos sidecar.

By having this new endpoint I envision Prometheus Operator calling it before scaling down the instance, and after this endpoint returns success we can just delete the Statefulset and its Persistent volume.

Prometheus operator has its own sidecar like the config-reloader. Why this feature cannot be implemented in another binary deployed as a sidecar as well? The code can be maintained in the Prometheus operator project.

My only concern about having this on projects like config-reloader is because this components will need to learn how to upload blocks to storage account.

Besides that I think might be a useful solution for people is not using Prometheus Operator as well, but I don't have any data about it to confirm it's just feeling 😅

from thanos.

As someone having to orchestrate "flushing" outside of prometheus-operator currently, I would definitely leverage a /flush endpoint in thanos. I'd also much prefer it as an in-built command than a separate container like config-reloader.

Once this functionality has been added to the Prometheus operator sidecar, there should be no difference in terms of UX compared to having the API natively?

But I think I am okay to add the new API. PR is welcomed.

from thanos.

There was some idea to run prometheus via libraries from within sidecar; that way we could build this properly. cc @SuperQ

from thanos.