New Dacha1 servers should not be able to start up if they are in a bad state about featurehub HOT 6 OPEN

Have landed along with a diagnostic API on Dacha1 FeatureHub instances. This is in 1.6.1-RC+

from featurehub.

So the Dacha1 protocol is such that a Dacha1 will become ready under two conditions:

(1) When it is told that the number of service accounts it now has in its cache is equal to the number of service accounts that is attached to each service account record, or it is told that there are no service accounts. Each PublishServiceAccount has the details of the service account AND the total number of service accounts.
(2) The same occurs for environments.

In a k8s cluster, a not-ready Dacha1 will not get traffic routed to it, so there is no issue with what the error response codes are.

I'm wondering if the issue is actually timed based. Explaining how this worked has made me think of a possible issue.

from featurehub.

I was checking to see if one dacha filling another dacha after a reset caused the original dacha to publish environment & service account "counts" with the wrong values, but they don't, if one dacha fills another, it resets the total number to what it has in its own cache correctly. Back to sq one.

from featurehub.

The code base as 3 dacha servers configured as test cases, i ran up two and then shut down the nats server and started the 3rd. The 1st two started complaining about lost connectivity and the 3rd wouldn't come ready. i started the nats server again and they recovered and the 3rd one advertised for its cache (when the health check was hit) and became ready (with the correct number of environments in the cache). So I'm at a bit of a loss.

from featurehub.

Similar issue that may happen after k8s has no resources left (cpu/ram/...) : when k8s is ready again :

• the features may not be updated correctly anymore (retired ones are still sent, changes are not sent)
• "API KEY does not exist." may happen with valid key (retries works with the same key)
• Everythings seems ok with the edge / admin / database

In order to be stable again, we need to stop all dacha and than start them again.

Not reproduced when k8s is not overloaded (thus it should not happen)... I was waiting to have more info before creating a issue.

from featurehub.

Yes, I expect that if something causes starvation it's going to be an issue. There is a PR that will let you do a full cache republish.

from featurehub.