Train issues about decima-sim HOT 3 OPEN

The training batches the graph in one episode (episode ends stochastically by reset_prob). For long episode with large graphs, this might overwhelm GPU's memory. Could you check the actual data size in a batch to see if this is the problem (we tried to do sparse graphs as much as possible but there might still be room for improving efficiency).

To try a smaller scale problem and get results quickly, you can try --num_init_dags 20 --num_stream_dags 0 to make it a batch case training (section 7.2 batch arrivals).

Hope these helps?

from decima-sim.

Thank you for the answer, but still, it runs slow/ out of memory. Can you please add the tensorflow version you use here? and also the parameters for the GPU: --master_num_gpu, --worker_num_gpu

from decima-sim.

I believe we developed the project with tensorflow version 1.14.0. With small batch job arrival experiment above, it shouldn't take that much memory and you can try just train it with cpu.

from decima-sim.