Lost data during compaction on Swift about mimir HOT 6 OPEN

Thank you for detailed report. As you point out, this needs to be fixed in Thanos first. We can keep this issue open until we import updated Thanos into Cortex.

Submitted by: pstibrany

from mimir.

Thanks for your report.

There's a thing I can't understand tho. You mentioned the compact has logged uploaded file on 01F16ZRT8TYA08VJQR1ZPCC5EP/index: this log gets logged once the upload successfully completes (from the client perspective) so, for the compactor, the index was successfully uploaded.

Submitted by: pracucci

from mimir.

From my understanding, an error happened in Upload(), but was only caught during the deferred CloseWithLogOnErr(). Upload() then falsely returned "success" to the compactor as it did not know that the Close() had failed. Based on the return value of Upload(), the compactor thinks the upload was a success, even though it failed.

Submitted by: ubcharron

from mimir.

this log gets logged once the upload successfully completes (from the client perspective) so, for the compactor, the index was successfully uploaded.

When using Swift, Upload returns success even if it fails when closing the writer :(

Submitted by: pstibrany

from mimir.

When using Swift, Upload returns success even if it fails when closing the writer

Oh, it's a Swift client issue! Let's fix it ;)

Submitted by: pracucci

from mimir.

This issue is resolved as of Thanos v0.21.0. Would it be possible to update the Thanos vendor pkg, or otherwise cherry-pick this the PR? (thanos-io/thanos#4218)

Submitted by: bcharron

from mimir.