[ingester] Ingester service state and lifecycler ring state not synchronized about mimir HOT 4 OPEN

@dimitarvdimitrov as the code stands currently, we shouldn't hit this issue, because (as you point out) all of the code that runs after the lifecycler is started should execute quickly, and not block the ingester's starting(). However, during development of the owned series service this behavior caused issues (#7087) and was confusing to navigate around.

So, this issue is less about fixing an active problem, but more about fixing non-deterministic behavior that can cause problems in non-obvious and can cause problems in the future.

from mimir.

shouldn't the lifecycler be started only after all of its submodules are in a Running state? What is holding up the startup of the ingester?

from mimir.

shouldn't the lifecycler be started only after all of its submodules are in a Running state? What is holding up the startup of the ingester?

Do you mean the ingester's submodules? There are several ingester subservices that require the lifecycler to be running first, at least according to the comment in that file. That said, maybe the comment is wrong and we could just move the lifecycler start to the end of starting()?

from mimir.

Do you mean the ingester's submodules? There are several ingester subservices that require the lifecycler to be running first, at least according to the comment in that file

ah, yes, that's what I was looking for. From those services only the ingestPartitionLifecycler has any starting procedure; the rest are timer services which start ~immediately. I still don't understand why that would hold up the ingester from starting for that long. Is it possible that some components haven't yet received the ring update that the ingester is shutting down and entering LEAVING state and still send it queries?

from mimir.