Comments (12)
ptnapoleon
commented on May 30, 2024
Hi @nikos912000, we've seen this too and we're hoping the 5.3.0 release (pushed 1m ago) will resolve it. We were seeing this problem (though I dont particularly understand why) when the disruption was attempting to delete the expired chaos pod too closely to when it was in the process of expiring and then cleaning itself.
from chaos-controller.
nikos912000
commented on May 30, 2024
Awesome, thanks for the prompt response @ptnapoleon :)
from chaos-controller.
nikos912000
commented on May 30, 2024
Hi @ptnapoleon,
I was testing the new version today. I see the following behavior and I'd like to confirm if this was the intention of the patch.
- The injector Pods end up in the
Failedstate. This now happens in the majority of the experiments. - Eventually the Pods are cleaned up once the Disruption expires.
Is this intentional so the Pods never end up hanging in the Terminating state?
from chaos-controller.
ptnapoleon
commented on May 30, 2024
Yes. I'll try to follow up with a PR that has the same behavior but without the injector pods ending up Failed
from chaos-controller.
nikos912000
commented on May 30, 2024
Got it, thank you 👍
from chaos-controller.
ptnapoleon
commented on May 30, 2024
I'm hoping #493 will resolve this issue
from chaos-controller.
nikos912000
commented on May 30, 2024
I'm hoping #493 will resolve this issue
Nice, thanks a lot. Would you like me to test that internally or you are planning to draft a release?
from chaos-controller.
ptnapoleon
commented on May 30, 2024
I will draft a release for you to test with
from chaos-controller.
ptnapoleon
commented on May 30, 2024
5.4.0 tagged, will be on dockerhub soon
from chaos-controller.
nikos912000
commented on May 30, 2024
Hi @ptnapoleon,
I tested 5.4.0 today and I see the same behavior, with the Pod in the Failed state.
It is not a critical issue as the Pod is eventually getting cleaned up, when the Disruption expires.
from chaos-controller.
ptnapoleon
commented on May 30, 2024
Interesting! It has stopped happening for us. Are your chaos pods failing to stop? We see the chaos pods into a successful Completed state, rather than DeadlineExceeded. Can you tell if they're having the correct duration passed to them from the controller?
from chaos-controller.
nikos912000
commented on May 30, 2024
Sorry for the delay @ptnapoleon, I missed this.
I tested this again today and it's not happening anymore. Not sure if something changed in our clusters as I only reproduced there.
Happy to close and thanks again!
from chaos-controller.
Related Issues (20)
- User Request: Debugging instructions HOT 3
- User Issue: Status not being a subresource causes issues with control planes HOT 5
- User Issue: Traffic surge once a packet drop network failure finishes HOT 6
- User Issue: CPU pressure does not consume 100% of the pods allocated CPU HOT 1
- User Request: store failures in the Custom Resource's status HOT 11
- User Issue: Security vulnerabilities flagged in Docker images with Go 1.16 HOT 4
- User Request: Release Dynamic Targeting behind a feature flag in controller HOT 4
- User Issue: Unable to terminate node level network experiments HOT 8
- Static Targeting example HOT 2
- User Request: Release cloudProviders behind a feature flag in controller HOT 5
- User Issue: Unable to gracefully terminate pods container HOT 9
- Guidance running CPU pressure experiments HOT 5
- User Issue / Suggestion: Controller arguments do not supersede config file HOT 2
- User Issue: Error pulling image with 7.19.0
- User Issue: CPU Pressure experiment could not inject the disruption successfully HOT 5
- Unable to dynamically target and experiments end on PreviouslyPartiallyInjected 7.22 HOT 4
- 7.26.0 Upgrade Issues - CrashBackLoop HOT 3
- 7.26.0 Upgrade Issues - InjectionStatus PreviouslyPartiallyInjected HOT 21
- User Request: Add support for tolerations on injector pods HOT 1
- User Issue: OnInit Chaos handler loop HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chaos-controller.