tanukkii007 / akka-cluster-custom-downing Goto Github PK

Hi !

I was experimenting with OldestAutoDowning strategy and stumbled upon a behavior when multiple nodes get into unreachable state at the "same" time - or within a "stable-after" window.

In that case OldestAutoDowning will down one node only, the events for other nodes will not get processed and things are stuck in "Leader can currently not perform its duties" state forever.

2018-04-11 15:23:04 [INFO ] [a.c.Cluster(akka://somecluster)]  - Cluster Node [akka.tcp://[email protected]:2552] - Leader can currently not perform its duties,
 reachability status: [
  akka.tcp://[email protected]:2552 -> akka.tcp://[email protected]:60202: Unreachable [Unreachable] (9),
  akka.tcp://[email protected]:2552 -> akka.tcp://[email protected]:60265: Unreachable [Unreachable] (8)
  ], member status: [
    akka.tcp://[email protected]:2552 Up seen=true,
    akka.tcp://[email protected]:60202 Up seen=false,
    akka.tcp://[email protected]:60265 Up seen=false
    ]

These are the settings:

akka.cluster.downing-provider-class = "tanukki.akka.cluster.autodown.OldestAutoDowning"

custom-downing {
  stable-after = 20s

  oldest-auto-downing {
    oldest-member-role = "master"
    down-if-alone = false
  }
}

Seems to me in this case
CustomAutoDownBase#downPendingUnreachableMembers will never get called in that case as its called from OldestAutoDownBase#onMemberRemoved only and somehow that will not happen.

How to reproduce:

• start node with "master" role
• start two other nodes not having "master" role
• kill the two nodes
• => one gets downed, the other one not, "Leader can currently not perform its duties" state forever

Is this a bug or a feature ?

Thx !

Ideally the akka dependency should be optional so that whoever uses the library doesn't have to use the same akka version as the one declared in this project

Would be nice if a release for akka 2.5.23 and scala 2.13 was released.

Now that Scala 2.12 is out it'd be nice to have a compatible published artifact.

See https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

Maven groupIds must not contain slashes, so using this from Maven/Java as currently hosted on Bintray requires some hacking. github.com/TanUkkii007 should ideally be changed to com.github.TanUkkii007.

We have there:
custom-downing.majority-auto-downing and custom-downing.majority-leader-auto-downing.
Shouldn't this be the same? I think we should stick to one naming.

Under reference.conf file there is defined section for majority-auto-downing but here https://github.com/TanUkkii007/akka-cluster-custom-downing/blob/master/src/main/scala/tanukki/akka/cluster/autodown/MajorityLeaderAutoDowning.scala we read down-if-in-minority and shutdown-actor-system-on-resolution from majority-leader-auto-downing.

It's clearly inconsistent.

Hi,

Just started to play around this very promising library, and following the instructions available here :
https://github.com/TanUkkii007/akka-cluster-custom-downing

I've setup my little akka app as follow :

akka {

    custom-downing {
       stable-after = 20s
       quorum-leader-auto-downing {
         role = ""
         quorum-size = 0
         down-if-out-of-quorum = true
       }
    }

    remote {
        netty.tcp {
              [... other stuff ...]
        }
    }
	
    cluster {
        downing-provider-class = "tanukki.akka.cluster.autodown.QuorumLeaderAutoDowning"
        
        [... other stuff ...]
    }

}

But got the following exception :

02/15-18:13:22<17794> Uncaught error from thread [activity-feeds-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[activity-feeds]
02/15-18:13:22<17794> java.lang.NoSuchMethodError: com.typesafe.config.Config.getDuration(Ljava/lang/String;)Ljava/time/Duration;
02/15-18:13:22<17794> at tanukki.akka.cluster.autodown.QuorumLeaderAutoDowning.downingActorProps(QuorumLeaderAutoDowning.scala:16)
02/15-18:13:22<17794> at akka.cluster.ClusterCoreDaemon.preStart(ClusterDaemon.scala:303)
02/15-18:13:22<17794> at akka.actor.Actor$class.aroundPreStart(Actor.scala:504)
02/15-18:13:22<17794> at akka.cluster.ClusterCoreDaemon.aroundPreStart(ClusterDaemon.scala:241)
02/15-18:13:22<17794> at akka.actor.ActorCell.create(ActorCell.scala:590)
02/15-18:13:22<17794> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:461)
02/15-18:13:22<17794> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:483)
02/15-18:13:22<17794> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
02/15-18:13:22<17794> at akka.dispatch.Mailbox.run(Mailbox.scala:223)
02/15-18:13:23<17794> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
02/15-18:13:23<17794> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
02/15-18:13:23<17794> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
02/15-18:13:23<17794> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
02/15-18:13:23<17794> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I'm using :

            <dependency>
                <groupId>com.github.TanUkkii007</groupId>
                <artifactId>akka-cluster-custom-downing_2.11</artifactId>
                <version>0.0.7</version>
            </dependency>

and this is what's part of my classpath in terms of "akka-related" libs :

akka-actor_2.11-2.4.16.jar
akka-cluster_2.11-2.4.16.jar
akka-cluster-custom-downing_2.11-0.0.7.jar
akka-cluster-tools_2.11-2.4.16.jar
akka-contrib_2.11-2.4.16.jar
akka-persistence_2.11-2.4.16.jar
akka-protobuf_2.11-2.4.16.jar
akka-remote_2.11-2.4.16.jar
akka-slf4j_2.11-2.4.16.jar
akka-stream_2.11-2.4.16.jar
akka-stream-experimental_2.11-1.0-RC4.jar
ssl-config-akka_2.11-0.2.1.jar

Any clue on why I'm not able to parse the "stable-afer" setting ?

Cheers,
Francesco

Somehow "DOWN itself when quorum is broken" test case of QuorumLeaderAutoDowningNodeThatIsUnreachableWithFailureDetectorPuppet spec fails frequently.
I guess cluster does not terminate after down itself when using FailureDetectorPuppet.

Current implementation is not based on auto-down-unreachable-after of AutoDown feature of akka cluster. Rather, it is semantically same as stable-after feature of split brain resolver.

Trying to use OldestAutoDown strategy and when self shutdown takes more than 10 seconds, cluster member doesn't terminate. Wish I could configure that parameter some way.

Hi, I'm trying to use your library but it seems there's an infinite recursion. I set up the downing strategy according to README but the application crashes immediately with

java.lang.StackOverflowError
	at tanukki.akka.cluster.autodown.OldestAutoDowning.downRemovalMargin(OldestAutoDowning.scala:13)

After inspecting the code, it really seems the overridden method downRemovalMargin is calling itself.