Stabilizing the signals feature - $SA 4M bounty! about superalgos HOT 4 OPEN

PR #5043 submitted to hopefully kill this issue (finally).

from superalgos.

As with every other bounty, people are free to chip in and pledge more SA to the bounty. I will update the title accordingly.

from superalgos.

I have been able to log one occasion of the trading task on the receiver side dying as follows:

The message points to an issue with writing the log file to disk, this leading to the task exiting - i.e. the logging engine may be the suspect for this specific issue. The previous error of not being able to download the signal package content occurs more regularly and is cleanly handled by SA.

I checked the docs of our logging engine Winston and found a few parameters which prevent the logging from exiting when such errors occur. A configuration change to logging including these parameters was submitted with PR #4753

This most likely isn't the root cause of all issues (sender side interruptions look behave different as per current reports). It may though be one piece of the overall puzzle.

from superalgos.

@BlaaSwe thankfully provided a bunch of log files after the recent Trend Soaring outages. The log file status for both outages I was able to analyze was identical.

This is the Trading Process log of the died process (last line after restarting the died task):

In successful process loops, the logging normally continues as follows:

Learning:
When hangups occur, the Trading Task seems to wait for an Event about a completed Data Process. This event though never arrives.

As a next step, I checked the logs of the Data Task the trading task was waiting for:

Learning:
The Data Task for which the Trading Task was waiting finished normally. The Execution Finished Event was correctly raised, but never arrived at the Trading Task.

Possible Root Causes (ideation):

• Event Server Outage (unlikely, as the other strategies on the same machine continued to work)
• Sudden Death of the Trading Task without leaving any trace in the logs
• Interrupted communication between Event Server Clients and Event Server (Websockets connections)

As Websockets Connections were already creating some headaches on the signal distribution side, they are my main suspect for now (despite sender and recipient running on the same machine here).

I have submitted PR #4833 to prevent undetected connection losses between event servers and clients. Let's see if this helps.

Further ideas based on above logging inputs are highly welcome!

from superalgos.