joesan / plant-simulator Goto Github PK

Right now the build pipeline just builds and pushes the changes to heroku. For running the application as a Docker image on Kubernetes, we need to build & push the image into Docker hub from where we will source the image for running in a k8s cluster.

Using KWH, MWH values for the power is done using Double datatype. Use the squants library to better abstract and represent measurement units.

http://www.squants.com/

In the DBServiceActor.scala, we pre-start it by reading the DBService method to fetch all active power plants. But this message is never received in the receive block of this actor as there is no match case to pattern match this message type.

This message needs to be processed in the receive block and the actor has to be context becomed so that successive messages are handled.

Here is the exception from the logs:

[INFO] [08/11/2017 13:57:59.390] [application-akka.actor.default-dispatcher-4] [akka://application/user/$a/$a] Pre-start DBServiceActor
[INFO] [08/11/2017 13:57:59.402] [application-akka.actor.default-dispatcher-4] [akka://application/user/$a/$a] done with pre-start DBServiceActor *****
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/$a/$a#-1046088460]] after [5000 ms]. Sender[Actor[akka://application/user/$a#-293595750]] sent message of type "com.inland24.plantsim.services.database.DBServiceActor$PowerPlantEvents$".
	at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
	at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
	at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
	at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
	at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:331)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:282)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:286)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:238)
	at java.lang.Thread.run(Thread.java:745)

Right now during CI, upon successful version tagging, we write the tag version into the plant-simulator-deployment project in its master branch. We need to avoid that and instead write it into the feature branch so that it runs via the Github actions that is defined in the plant-simulator-deployment project that runs for all feature branches.

We need a Kafka broker to which we can write signal data!

Currently, there is now way to simulate an OutOfService and ReturnToService scenario. We need a HTTP Endpoint to simulate this. We additionally also need to alert when this happens!

When pushing a tag, the tag should be extracted using some environment variables. This could currently now be done as there is no GitHub function / feature that could be used to access the tag name. Fix this!

In the signals that we get back from the WebSocket endpoint, the PowerPlant ID is missing! Please add it:

{
"activePower" : "20.0",
"isOnOff" : "false",
"isAvailable" : "true"
}

Currently, the application runs raw without any container. We probably need to get into a container-first approach where we run the services from within a container.

So explore on Kubernetes and see how we can spin a kubernetes setup on some environments!

When the PowerPlant is in dispatched state, we could add a SetPoint signal along with the telemetry! This could be useful for the UI to show an additional plot for the SetPoint!

We need a timer to check the latency on the calls to the database. Currently every call that involves a database lookup takes seconds. We need a way to track this.

Codahale library need to be integrated!

There are 2 problems with the current JSON formatters:

1. The validate and fold in the controller is not working as expected. Have a look at this post:

https://stackoverflow.com/questions/48855193/play-framework-strange-behavior-when-parsing-json-body

2. When using FiniteDuration, make sure to use just the length of the FiniteDuration and avoid using the TimeUnit in the JSON field. So it should be as 2 instead of 2 seconds

Currently, the setPoint is added to the telemetry signals only during dispatch, but we need it to be shown always. I have implemented this for RampUpType, we need to do the same for OnOffType.

When ReturnToNormal for a RampUpType PowerPlant, we need to mimic reality and have some randomness when the PowerPlant is going down to its minPower!

When dispatching a RampUpType PowerPlant, we need a mechanism to send ReturnToNormal signal so that the PowerPlant comes to its default state! Currently, there is no way to do this! This is a bug!

Currently the telemetry signals are emitted without a Timestamp! We need a Timestamp on this data! Make sure no tests fails after adding the Timestamp! Do this for both the PowerPlant types.

Travis CI behaves mysteriously. Occassionally it stopped to report build status to GitHub and this is annoying as I have automated PR merges configured using Mergify. So move over the GitHub actions.

Currently no unit testing is available for the Play controllers. We need some unit testing to be added so that we can have a good test coverage!

The currently existing WebSocket endpoint for Alerts and Events has the ability to filter for PowerPlant Id, but that does not seem to work currently. Even after passing in the PowerPlantId as a request parameter, the filter seems not to respect it and we get to see events and alerts for all the other PowerPlant's as well!

We need to create a H2 database for load testing! This database should contain at least 100000 PowerPlant's. The script should create a db file which can then be used as the database and then to start the plant-simulator application with this database. This way, we can judge if the plant-simulator can cope up with those many PowerPlant's!

Currently the DB scripts to set up the database is contained in the following repository:

https://github.com/joesan/database-projects/tree/master/power-plant-simulator

This at the moment contains the DDL and DML for MySQL which is hosted at https://www.gearhost.com/

Likewise we need to prepare the same for Postgres and perhaps also identify if there is a free cloud hosting for Postgres!

Additionally, an automatic mechanism to run the script against the hosted service should be identified.

A WebSocket endpoint that contains alerts and events should be created. This endpoint should be parameterized such that we can filter for events and alerts based on the PowerPlantId.

We need to come up with a load testing setup with which we can test and assert the performance of the application.

Step 1:

1. We need to create 100,000 PowerPlant's in the database (Write a scala script that can create those many insert statements and insert them in the database)

2. We need to then run this application against this 100,000 PowerPlant's and see how the application performs

Step 2:

1. We need to push the events, alerts and the telemetry signals to a Kafka topic (one topic each for event, alert and telemetry)

2. Open a WebSocket endpoint for one or a couple of assets and steer it and see if the application runs smoothly! If yes, we are good!

For all the PowerPlants that are configured in the database, we need to apply a Schedule. This Schedule should then be used to calculate the run plan for each of the PowerPlant based on the available capacity and spit out results that can tell if there is a surplus or if there is a deficiency.

This surplus or deficiency should then be fed as a trade in the Blockchain!

Currently using Play Framework 2.5.9, upgrade to Play 2.6.11

See here for reference: https://www.playframework.com/documentation/2.6.x/Migration26

There is this insertOrUpdate option in Slick that can either do an insert or an update. We need some mechanism to use just one service method to do both these options! Currently there are 2 seperate methods in the service to do this.

When opening a WebSocket endpoint, the events and alerts that I get are not in the proper order! For example., when a PowerPlant starts, it should start with an Init and then go to Active, so the order or events that I receive with the WebSocket endpoint should be Init and then Active! Please have a look and fix!

Alerting and Eventing should be added to OnOffType PowerPlant, similiar to RampUpType.

We need some Scala formatter that can format the contents before comiting!

Check what state the PowerPlant should be when coming back from OutOfService? Should it be set to Active? If yes, then currently this is not the case. The State is not going to Active upon ReturnToService

Currently all the PowerPlant actors send alerts and events to the EventStream Actor. Each PowerPlant Actor has a reference to the EventsStream actor which is given when the PowerPlant actor starts up.

Now when this EventsStream actor gets killed for whatever reason, the PowerPlant actors could not send any more events and alerts. We of course have Akka supervison, which means the EventsStream actor will get resurrected. We now need to make sure with some tests that after the EventsStream actor gets resurrected by the supervisor, all the PowerPlant actor instances can continue sending events and alerts.

Currently there is an endpoint to fetch all the available PowerPlant's from the database. This endpoint need to be paginated such that only a certain amount of data is fetched upon every request!

The Endpoint is /powerPlants?onlyActive=false&page=1

No Eventing and Alerting seem to happen for the OnOffType PowerPlant! I do not see the events and alerts when I open the WebSocket connection!

We need a swagger API documentation to test the API's!

We need some alerting mechanism when an unwanted abnormal behaviour occurs. For example., when we send a dispatch command for a RampUpType PowerPlant and this dispatch command says that the PowerPlant has to dispatch 900kw of power, but this PowerPlant only has 800kw capacity, we curtail the setpoint to 800 and just log a warning message. Such events should be captured as alerts and such alerts should be piped to a WebSocket stream that can be subscribed by any other component.

When the PowerPlant is in OutOfService, it is not available for service. So it should not respond to any messages from outside! We should rather bring it back to Service by issuing the ReturnToService command and then we can use it for dispatching!

Currently this is possible to dispatch a PowerPlant that is OutOfService! Fix this!

Using the Tagless final design pattern, refactor the DBService layer so that it can work against any implementation. Right now it works only with a Future API. By implementing the Tagless final design pattern, we can make it to work with Task or any other API as a matter of fact!

Currently there is no alerts emitted by the StateMachine when a PowerPlant goes into OutOfService or comes bak to ReturnToService

Currently the CRUD methods in the PowerPlantController is a bit messy. There are a lot of nested pattern matching clauses and statements. This should be simplified. One way is to introduce a Service layer between the controller and the DBService where we could move all the nested checks into this. By having the tagless final, we can probably do flatmap like operations instead of the nexted case blocks.

Currently when a RampUpType PowerPlant is up and running, there is no randomness in the power values. To be more naturalistic, it would be ideal if we can emit varying signals with some randomness.

Once this randomness is built in, we should then pipe the signals from the PowerPlant to a WebSocket endpoint. The frequency at which the signals are emitted should be configured!

We need some mechanism to persist the state of a PowerPlant actor to the database. This database could either be a MongoDB or some redis cache, but this cache should be persistent such that we can use this to resurrect an Actor. We could also use Akka persistence for this purpose. Do some analysis on this!

OnOffType PowerPlant when in dispatched state seems not to ReturnToNormal when it receives such a command. Please fix this!

Currently when running the application, we need a running instance of the database. This could be simplified by having a local H2 database setup.

The H2 Setup exists only for the unit tests, but we need one that we could use for the application!

Currently there is no way to do CRUD operations on a PowerPlant. We need an endpoint each to do the following:

1. Add - /powerPlant/add - Adds a new PowerPlant to the database

2. Update - /powerPlant/:id/update - Updates an existing PowerPlant for the given id

3. Delete - /powerPlant/:id/delete - Performs a soft delete (set the isActive flag to false)

4. Read - /powerPlant/:id/details - This already is implemented

We need some mechanism using travis that can automatically deploy to Heroku after every successful build!

Slick has set some of its Logging to DEBUG and this is pretty annoying. This must be turned off and set to INFO!

Right now only one instance of the simulator can be run. As we build StateMachines using Akka only one instance can be runnable. We need to however make the StateMachines redundant, meaning that we have to be able to spin up another instance on demand or when the application that runs the PowerPlant goes down.

This can be done using Akka Persistence and Akka Cluster Sharding. Refer to the following documentation:

https://doc.akka.io/docs/akka/current/cluster-sharding.html

https://doc.akka.io/docs/akka/current/persistence.html

We need to push signals to Kafka topic! So we need a Kafka server setup.

We need an Endpoint to which we can listen for the PowerPlant signals emitted by the PowerPlant. This Endpoint should be a WebSocket endpoint with BackPressure! This Endpoint should also be filterable by the PowerPlant ID!