User Issue: Injector Pod stuck when executing kernel panic disruption about chaos-controller HOT 9 CLOSED

OR, in the injector, remove the finalizer before writing to sysrq and re-add the finalizer on failure (since it is the only part of the code that will still be running if the injection fails) so we can catch that later

I like this option most, I'll draft up a PR for Nikos to test

from chaos-controller.

Oh I see. This isn't a problem for us because the node gets replaced on any nodeFailure disruption, which is why the only way we handle it is to check if the target node is gone, and if yes, go ahead and clean up the stuck pod.

The only idea for a fix I have for this is to go ahead and remove the chaos pod finalizer from within before we create the kernel panic, though it runs the risk that the disruption doesn't occur

from chaos-controller.

Interesting.

I'm curious why the node gets replaced for you.
Does its health-check fail and auto-scaling kicks in?
If this is the case, what is the difference between the 2 node failures for you from a behavior perspective?

I think it is sensible to remove the finalizer once the Disruption expires?

from chaos-controller.

I think it is sensible to remove the finalizer once the Disruption expires?

Hmm, for node failures specifically, that seems fine? There's nothing to clean, really. @Devatoria ?

If this is the case, what is the difference between the 2 node failures for you from a behavior perspective?

Almost nothing, afaict

I'm curious why the node gets replaced for you.

I think it's cloud provider health checks failing

from chaos-controller.

@ptnapoleon Would it be worth adding a bit more logic on node disruption cleanup? I see multiple options here:

• in the controller, ensure the node is NotReady if the chaos pod exit code is not 0 (we can conclude that the node is disrupted) before removing the finalizer
• OR, in the injector, remove the finalizer before writing to sysrq and re-add the finalizer on failure (since it is the only part of the code that will still be running if the injection fails) so we can catch that later

Wdyt?

from chaos-controller.

I'm testing #503 . It seems simple and safe. Altering the finalizer logic by having chaos pods mutate their own finalizer was causing reconcile issues in the controller

from chaos-controller.

This seems to be working locally and on staging for us, let me know if it's sufficient for you!

from chaos-controller.

Thanks Philip!
I will test this today in our clusters and will let you know.

from chaos-controller.

@ptnapoleon perfect, this works as expected. Thank you Philip.

from chaos-controller.