genixpro / kwola Goto Github PK

Traceback (most recent call last):
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/tasks/RunTrainingStep.py", line 646, in runTrainingStep
results = agent.learnFromBatches(batches)
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/components/agents/DeepLearningAgent.py", line 1499, in learnFromBatches
"computeRewards": True
File "/home/bradley/venv/lib64/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/bradley/venv/lib64/python3.7/site-packages/torch/nn/parallel/distributed.py", line 442, in forward
self._sync_params()
File "/home/bradley/venv/lib64/python3.7/site-packages/torch/nn/parallel/distributed.py", line 515, in _sync_params
self.broadcast_bucket_size)
File "/home/bradley/venv/lib64/python3.7/site-packages/torch/nn/parallel/distributed.py", line 485, in _distributed_broadcast_coalesced
dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 1800000ms for recv operation to complete

Running Kwola on dockerized ADAE project. The run produced bugs folder and videos, at somepoint this console error was thrown and the run stopped after.

Running kwola again on the run folder produced, this error was looping without stopping.
last log output:
2020-04-26 21:13:43.601079 [15220] TaskProcess: Waiting for input from stdin
2020-04-26 21:13:43.601216 [15220] Running process with following data:
{
"configDir": "kwola_run_014",
"trainingSequenceId": "kmbo",
"trainingStepIndex": 0,
"gpu": null
}
2020-04-26 21:13:43.602209 [15220] Starting Training Step
2020-04-26 21:13:48.182703 [15310] Starting initialization for batch preparation sub process.
2020-04-26 21:13:48.188535 [15310] Starting loading of execution trace weight datas.
2020-04-26 21:13:48.190342 [15310] Finished loading of weight datas for 0 execution traces.
2020-04-26 21:13:48.190381 [15310] Finished initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.442085 [15310] Resetting batch prep process pool. Cache full state. New workers: 1
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.465900 [15309] Starting initialization for batch preparation sub process.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
2020-04-26 21:13:48.503050 [15309] Starting loading of execution trace weight datas.
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue
prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue

Terminal:

"success": true,
"testingStepId": "d4if_testing_step_6"
}
Traceback (most recent call last):
File "/home/bradley/venv/bin/kwola", line 8, in
sys.exit(main())
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/bin/main.py", line 181, in main
TrainAgentLoop.trainAgent(configDir)
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/tasks/TrainAgentLoop.py", line 259, in trainAgent
runMainTrainingLoop(config, trainingSequence, exitOnFail=True)
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/tasks/TrainAgentLoop.py", line 208, in runMainTrainingLoop
raise RuntimeError("One of the testing / training loops failed and did not return successfully. Exiting the training loop.")
RuntimeError: One of the testing / training loops failed and did not return successfully. Exiting the training loop.

It completes one round of testing but doesn't continue.

Code attached in text file

Untitled document.pdf

The number of changes we have to make is relatively small as there is only a single line of code at this time that is attributed to not work on older Python: that's JSRewriteProxy line 98

Traceback (most recent call last):
File "/home/bradley/venv/lib64/python3.7/site-packages/kwola/tasks/RunTrainingStep.py", line 356, in prepareAndLoadBatchesSubprocess
differenceRatio = abs(traceWeightData['weight'] - sampleRewardLoss) / traceWeightData['weight']
ZeroDivisionError: float division by zero

Might have to do with the functionality of the app when it comes to looking at invoices. There might be a way to click off of it? Or it might be a weird spot to click off it too