romalev / consul-cluster-manager Goto Github PK

https://github.com/romalev/vertx-consul-cluster-manager/blob/master/.travis.yml - enable tests.

In order to achieve this consul agent has to embedded within test suite.

Investigate:

Get rid of the cache manager - instead figure out the way of how to plug caches directly into the cluster async map/multimap.

Round robin load balancing seems to be not working between two subs of the save event bus address - this has to be addressed.

Clearly the following is not correct :
CacheMultiMap is : '{
"vertx-consul" : {
"ids" : [ ],
"empty" : true
}
}'

Failing node (service) keeps residing within Consul for one minute since the time its health check has failed. Now when new node is about to join the cluster and this new node has same port as the node that's failing, then -> failing node theoretically will become available again (in case new node with the same port gets eventually registered).

This behavior should abounded.

New consul-based clustered hash map should be built.

Document every little detail.

Implement the basics of AsyncMap<K, V>.

See : https://github.com/romalev/vertx-consul-cluster-manager/milestone/2
Goes to first prod release:

• cm configuration! right now it looks a bit messy - come up with the proper way of cm configuration.
• subs are being properly load balanced.
• subs are ephemeral.
• hainfo map is NOT ephemeral.
• verify node leaving the cluster behavior.
• verify node joining the cluster behavior.
• distributed lock and distributed counter behavior.
• TCKs against this CM implementation must pass.
• HA.
• Run findbugs against this CM implementation.
• Run the CM within docker image.
• Logging.
• Perform some cleanup once manager is stable.
  Perform load testing on given cluster manager.
  Code review to be performed by vertx experts.

Goes to second prod release. To be discussed with vert.x experts:
Restore node's caching data in Consul. Suppose consul agent was down in the cluster for some time (meaning that data consul was holding potentially was lost) and it is up and running again - verify whether data located in the local caches are consistent with the data located within consul agent kv store.

HA must be working.

Check if the current implementation of port & host dynamic assignment is correct. See how port and host gets assigned for even bus subscribers.

Once the node is down - all its appropriate data in consul has to be cleaned up (service itself + appripriate distributed map KV pair(s)).

Consider:

• shutdown hooks;
• tcp consul checks;

Steps to reproduce:

1. Launch consul agent.
2. Launch "bob" service.
3. Shut down console agent.

• now the multi map cache is empty.

4. Launch consul agent again and see the log output.

2018-10-09 09:52:03 [vert.x-eventloop-thread-5] ERROR i.v.c.e.i.c.ClusteredEventBus - Failed to remove sub
java.lang.IllegalStateException: Result is already complete: succeeded
at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:87)
at io.vertx.spi.cluster.consul.impl.ConsulAsyncMultiMap.lambda$remove$8(ConsulAsyncMultiMap.java:94)
at io.vertx.core.Future.lambda$compose$1(Future.java:265)

seems to be a bug.

Consul has an interesting behavior describer here - hashicorp/consul#1172. I think this somehow should be taken into account - it has to be investigated.

Addressed in:

• #62
• #61
• #60

Apply cache to async map + cover async map by integration tests.

Currently given implementation of this cluster manager contains lots of code that first reads an entry and then update or remove it - sort of CAS operation which is not synchronized by consul agent.

Imagine a scenario where entry is being read by Alice in order to get replaced (or removed or updated) and in the meanwhile Bob wants to update same entry and he actually does. Bob's write takes place before Alice proceeds which means Alice's read value is not consistent anymore with central storage. Question: Are we allowing Alice's write forcibly to take place or should we come up with something like

• CAS;
• entry locking;
• applying consul transactions;

to abort Alice's write ? It has to be investigated.

Given following consul restriction :

TTL value (on entries) must be between 10s and 86400s currently. Invalidation-time is twice the TTL time - this means actual time when ttl entry gets removed (expired) is doubled to what you will specify as a ttl.

Custom handler should take place in order to satisfy vertx cluster managing SPI.

Current implementation of distributed maps has to be reviewed by taking into account:

• cas : http://alesnosek.com/blog/2017/07/25/check-and-set-operation-and-transactions-in-consul/
• caching - concurrent hashmap has to be added to async map and multimap.

So far we use dummy (V) valueFromConsulKVStore which is not sufficient.

ITs must be written to cover as many edge cases as possible.

Details :
testsuite.txt

https://github.com/romalev/vertx-consul-cluster-manager/blob/b0eae3890b905148e13860667fbb7c845895d3dc/src/main/java/io/vertx/spi/cluster/consul/ConsulClusterManager.java#L455

InetAddress.getLocalHost().getHostAddress() - is executed to get node's address.
How about if node has multiple network interfaces or application runs inside a docker container?

Relates to #44

• most of logging has to be removed when consul cluster manager is more or less stable.
• everything has to be documented in javadocs.

Now managing sessions within this CM implementation happens in two places :

• in consul abstract map.
• in consul cluster manager.

Define a way to have one SessionManager.

Figure out the way of how properly to get the manger initialized. Come up with ConsulClusterManagerOptions or smth like that.

Imagine a scenario where map entry with ttl already present within consul kv store. Now the vertx node is about to update given entry by wanting to put new ttl on it. Currently new session is gonna get created but won't be bound to existing entry we want to update ttl on.

AsyncMap comes with contract to have ttl on entry - this has to be addressed.

• Perform load testing on given cluster manager.
• Code review to be performed by vertx experts.
• Restore node's caching data in Consul. Suppose consul agent was down in the cluster for some time (meaning that data consul was holding potentially was lost) and it is up and running again - verify whether data located in the local caches are consistent with the data located within consul agent kv store.
• #78
• #42

Suppose consul agent was down in the cluster for some time (meaning that data consul was holding potentially was lost) and it is up and running again - verify whether data located in the local caches are consistent with the data located within consul agent kv store.