myawnhc / bankinabox Goto Github PK

Currently the minimum Java version required is 11, which does not align with Hazelcast's offering.
Additionally, not a large developer population is on 11 yet and forcing them is likely to be discouraging.
This should be brought down to Java 8.

Ran the demo as documented with one exception - 2 IMDG members instead of 3, and ran into the below error after 7 mins run:

SEVERE: [localhost]:5710 [JetInABox] [3.1] Execution of job 'AdjustMerchantTransactionAverage', execution 245a-d151-63fc-9ae2 failed after 408,673 ms
com.hazelcast.jet.JetException: Exception in ProcessorTasklet{AdjustMerchantTransactionAverage/Draw Transactions from preAuth map#0}: com.hazelcast.jet.JetException: Null key from key extractor, edge: between("Draw Transactions from preAuth map", "Aggregate average transaction amount by merchant-prepare").partitioned(?)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:315)
	at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:803)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.run(TaskletExecutionService.java:289)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at ------ submitted from ------.(Unknown Source)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:126)
	at com.hazelcast.spi.impl.AbstractInvocationFuture$1.run(AbstractInvocationFuture.java:251)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Caused by: com.hazelcast.jet.JetException: Null key from key extractor, edge: between("Draw Transactions from preAuth map", "Aggregate average transaction amount by merchant-prepare").partitioned(?)
	at com.hazelcast.jet.core.Edge$KeyPartitioner.getPartition(Edge.java:588)
	at com.hazelcast.jet.impl.execution.OutboundCollector$Partitioned.offer(OutboundCollector.java:193)
	at com.hazelcast.jet.impl.execution.OutboxImpl.doOffer(OutboxImpl.java:241)
	at com.hazelcast.jet.impl.execution.OutboxImpl.offerInternal(OutboxImpl.java:141)
	at com.hazelcast.jet.impl.execution.OutboxImpl.offer(OutboxImpl.java:93)
	at com.hazelcast.jet.core.AbstractProcessor.tryEmit(AbstractProcessor.java:310)
	at com.hazelcast.jet.core.AbstractProcessor.emitFromTraverser(AbstractProcessor.java:410)
	at com.hazelcast.jet.core.AbstractProcessor.emitFromTraverser(AbstractProcessor.java:423)
	at com.hazelcast.jet.impl.connector.StreamEventJournalP.emitResultSet(StreamEventJournalP.java:209)
	at com.hazelcast.jet.impl.connector.StreamEventJournalP.complete(StreamEventJournalP.java:194)
	at com.hazelcast.jet.impl.execution.ProcessorTasklet.stateMachineStep(ProcessorTasklet.java:367)
	at com.hazelcast.jet.impl.execution.ProcessorTasklet.call(ProcessorTasklet.java:232)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.runTasklet(TaskletExecutionService.java:308)
	at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:803)
	at com.hazelcast.jet.impl.execution.TaskletExecutionService$CooperativeWorker.run(TaskletExecutionService.java:289)
	at java.base/java.lang.Thread.run(Thread.java:834)

MerchantAvgTxnCheck is EntryProcessor based from the older architecture; now that RuleExecutor architecture is working it needs to be migrated into a new MerchantRuleSet

Objects such as Account are IdentifiedDataSerializable

Others such as RuleSetExecutor and CreditLimitCheck are java.io.Serializable

All should ideally be IdentifiedDataSerializable, or at least those that are performance impacting

Currently configuration is done through a properties file, which is an antiquated design.

Change to use yaml (probably best) or json (or xml but probably distant third choice) for configuration.

Add a ruleset and rules showing location based fraud detection.

Currently plan is to use geohash to represent various locations (merchant location, customer home location, mobile device location)

Currently, the preAuth map listener writes the same transaction to multiple queues in order to pass it to different RuleExecutors for processing. This is an expensive duplication - maybe not too bad if executors are keeping up, but will quickly become an issue if executors fall behind.

A pub/sub model using ReliableTopic seems preferable.

The current RuleExecutor is built around CompletableFuture and works well with the IQueue.take mechanism for input; it's not clear how to adapt this to the event-driven onMessage style of a topic.

Jet should be running in client-server mode, but was throwing an exception about not finding the service so I've reverted to embedded for now.

There is also an issue with Jet running client/server in Kubernetes that is fixed in 4.0, so we may leave this embedded until 4.0 or a fix for that issue is added to 3.12.x.

Current design: Transaction generator runs concurrently with the processing, generating random transactions that are written to the preAuth map. So transaction generation is competing for the same CPU cores that the processing needs.

Preferred design: Pregenerate transactions and store them (database or whatever); have an ingest mechanism that can vary in rate to drive different loads for testing. Rate should be variable to simulate spikes as expected in real-world workloads.

Create a RuleSet and rules that utilize a machine learning model to evaluate transactions and flag potential fraud.

Allow multiple run modes - Demo and Benchmark

Various things will work differently depending on run mode.

• Demo mode is meant to run continuously (for example, all day at a trade show)

• Benchmark mode is meant to run for a fixed length and collect statistics that can be used to compare different design alternatives.

• Demo mode needs to be more concerned about out of memory issues and manage the working set carefully via map eviction, etc.

• Demo mode supports a Grafana dashboard that isn't needed for benchmark

Imported dashboard into Grafana and started the demo. There is no activity in Grafana and all dials are sitting idle. On the other hand, Management Centre shows activity.

See attached screenshots of both Grafana and mancentre from the same time.