Program Hang in training stage about gazeml HOT 5 OPEN

@WuZhuoran
So did you encounter this also? Did you kill the process normally? Actually, I am not quite sure that I have finished training or not. If I kill it, and it means that I have to restart training over and over again.

@keishatsai I did encounter before. But at most time, I can stop the process normally.

from gazeml.

I am not sure if this program can run on Windows10.

But how many UnityEyes images do you use?

After you stuck after saving point, did you check if the program is still running? Did you check nvidia-sim whether the elg model is still using full gpu?

Exiting thread problem can be solved by killing process.

from gazeml.

Hi @WuZhuoran ,
Thanks for replying. It did use nearly full gpu (7G over 8G memory) while it hung. I made nvidia-smi to check every 5 secs, so I know.

Currently, I prepared 7524 images to train.

What do you mean by " Exiting thread problem can be solved by killing process. " ?

from gazeml.

I mean, if you found you cannot stop the process. You can use command:

kill -9 ${PROCESS_ID}

to exit the process.

from gazeml.

@WuZhuoran
So did you encounter this also? Did you kill the process normally? Actually, I am not quite sure that I have finished training or not. If I kill it, and it means that I have to restart training over and over again.

from gazeml.