Customizable HTTP Server Actions (Sink) Retry Policies about emqx HOT 2 OPEN

Hi @jandillmann
I think 503 is a good indication of temporary error, we could even consider it a bug if retry is not done for this error code.
In general, one should never retry 500, however I agree that we should probably add an option like "Always Retry (N times)".
i.e. retry attempts counter should not exceed N.

In stream/event sourcing, a so called poison-pill message is very common to happen, which typically means such a message may cause server to raise an exception or even crash.
When there are multiple consumers (http servers in case of HTTP sink) behind a load-balancer (or gateway), without a limit, the retries of a poison-pill message may eventually bring down all the consumers.

from emqx.

Yes, I agree, 500s should not be retried indefinitely. But a configurable small number of retries might be ok for my use case – it is usually only once per day or even less that a request does not go through. Retrying 429 and 503 sounds like a reasonable choice for me, too.

As for KrakenD returning a 500 instead of a more suitable 503 (or have it configurable), I will create an issue there.

from emqx.