Test Retry mechanisems about term-project-cloudriven HOT 1 CLOSED

Based on my observation, when a system faces some percentages of error, with help of the retry function the cluster can recover. And from the client's point of view, it works normally.
From Istio Retries documentation: Default retry is hardcoded and its value is equal to 2. It means any circumstance, it will try two times at least.
From this experiment, We simulate the situation where the db microservice faces some problem and rejects some percentage of requests by error code 500. The fault injection has been done with help of istio. On the other hand, we tried to use the retry function to recover the system.
It is clear that the retry function can help to recover the system, but the efficiency of the retry function depends on the percentage of failure and traffic of the network. The details exploration has been done and reported in this report (#29).
but these retries can make problems of congestion inside the cluster which can be controlled by the circuit breaker. (#33).
Interesting facts, regarding the number of retries:
I have injected 100% fault in db so I expected to see that s1 retry to send a request based on the number of retries. But my observation was different, In the beginning, there is no retry configuration, but istio as a default configuration retry to send the failed request again, because of that failed requests are 20, two times of origin request which was 10. As you can see in the diagram the number of requests is not following the number of retries. It seems, that there are mechanisms that prevent the retry function to go linear when the number of retries increases.

01.pdf

from term-project-cloudriven.