User Request: store failures in the Custom Resource's status about chaos-controller HOT 11 CLOSED

@clairecng @ptnapoleon thanks for looking into that.

Claire, that sounds good. I had a feeling the Disruption does not get created at all if it fails that validation, isn't it?

On a similar note, I started a discussion on how visibility/troubleshooting could be improved.

from chaos-controller.

@nikos912000 The disruption is already created when passing through the ValidateCreate method of the webhook, and is deleted when an error is returned.

from chaos-controller.

I assume the events that we record on the Disruption object are lacking, but could you expand on why? I do know that the API for reading events on a k8s object is extremely inconvenient compared to reading the status. We could do this, but at least for our users (and we don't use CD for disruptions here too often), they are always checking feedback via kubectl describe

from chaos-controller.

I need to do some digging on the events and on how to get them through our control plane. I'll post an update on this 👍

Do you know if we can somehow get the errors from the validating webhook? Since no Custom Resource gets created in that case we cannot have these errors in the status or in events isn't it?

from chaos-controller.

As far as I know, the validating webhook should already be returning errors, if it finds any? We try to catch every error that can be caught upfront within the webhook

from chaos-controller.

The validating webhook does return errors. However, I am not sure how we can get them from a client.

For instance, when I apply a Disruption with an invalid Service name, the webhook errors as expected from the codebase.

But this error message is not logged in the controller and no events are sent. It is only returned as a response to kubectl apply:

Error from server (1 error occurred:
	* Spec: Network: the service specified in the network disruption (chaos-engineering-framework/dem) does not exist

): error when creating "network_egress_drop_service.yaml": admission webhook "chaos-controller-webhook-service.chaos-engineering-framework.svc" denied the request: 1 error occurred:
	* Spec: Network: the service specified in the network disruption (chaos-engineering-framework/dem) does not exist

I guess this response comes from the controller-runtime's webhook handler.

from chaos-controller.

@Devatoria @clairecng opinions on adding this?

from chaos-controller.

If we want to implement this, we would need to create the disruption even if there is an error detected by the disruption webhook validator and let it sit without injection, which would need to be either updated or deleted by the user afterwards. Waiting for @Devatoria's opinion on this, as I don't think I have sufficient knowledge to know if this could be very bad or not.

If it's ok to change this behaviour, I don't think it should be too hard to either add events to the disruption (which could then be broadcasted to all our notifiers) or store failures in the disruption's status! I would have no problem with adding this bit.

If it's not ok to change this behaviour, we could send an event to the chaos-controller pod eventually, however I'm not sure it's a good idea... But we can still log it, if this could help?

from chaos-controller.

After some time thinking/testing, we could send an event on the validation here, even if the disruption is deleted just after that. Using the kubernetes API, it could still be accessible by listing all events and doing a little bit of filtering. (for example: kubectl get events --field-selector involvedObject.kind=Disruption).

@nikos912000 Would this be a good solution for you?

from chaos-controller.

👋 Is there still a need for such a feature @nikos912000 or is it solved? Or should we discuss that more generally in the discussion you linked?

Sorry for the late answers on most issues, we spent a good amount of time on internal dev efforts for the past months but we're now going back on the chaos-controller more actively and soon with some new features!

from chaos-controller.

👋 @Devatoria this is no more needed from our side - we can close it.
For context, we added links to DataDog Events and logs in our Continuous Deployment platform and so far it has been sufficient.

from chaos-controller.