Split Brain on HTTP requests for features flag requests : Dacha restart fixes about featurehub HOT 5 CLOSED

Thanks, please close.

from featurehub.

Heya - could you consider upgrading to 1.7.0 please? We went on a big bug hunt in that version for edge cases for Dacha2 particularly around webhooks and there were some concurrency issues we managed to iron out. We will however add a longer running e2e test to the suite that checks this - we will need to add some test APIs that let us knock out the dacha2 instances so they lose state. Are you running your eks on fargate (which will use karpenter) or are you using something like karpenter yourselves?

That stack trace indicates that your Prometheus server is terminating the GET request early so while the server is writing out metrics it is getting a socket closed fault (so it logs it).

from featurehub.

Hi @rvowles , thanks for the answer, updated from 1.6.0 to 1.6.3 two weeks ago, the jump to 1.7.0 on a mostly production system , was worried too big of a jump.
After discussing with my team:
Was running on EKS EC2 instances that were instance type t3.medium with replica of 2 for dacha and edge.
Karpentar is on our road-map, using Autoscaling Groups.
Modified this now to instance type m5.large with 3 replica's for dacha and edge, and added topologySpreadContraints : for Zones (AZS),
Noticed Network IN(Bytes) and Network OUT(Bytes) increased by a little bit.
Perhaps application is sensitive to network issues.
Will plan update to 1.7.0

from featurehub.

1.7.1 is already out by the way.

from featurehub.

@chkp-yaakovg if you need any further help - feel free to leave a discussion item or ideally jump onto our Slack!

from featurehub.