synxlin / deep-gradient-compression Goto Github PK

Traceback (most recent call last):
File "train.py", line 415, in
main()
File "train.py", line 191, in main
loader=loaders['test'], split='test')
File "train.py", line 315, in evaluate
for step, (inputs, targets) in enumerate(loader):
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in next
data = self._next_data()
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 989, in _next_data
return self._process_data(data)
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
data.reraise()
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in pin memory thread for device 0.
Original Traceback (most recent call last):
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 31, in _pin_memory_loop
data = pin_memory(data)
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in pin_memory
return [pin_memory(sample) for sample in data]
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in
return [pin_memory(sample) for sample in data]
File "/home/sr6/jinkyu.yim/vir_python3.7/lib/python3.7/site-packages/torch/utils/data/_utils/pin_memory.py", line 47, in pin_memory
return data.pin_memory()
RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory

python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown

Hello,

I wanted to try out your code and came across an issue regarding pytorch dependencies.

I installed all the requirements in a fresh conda environment with python 3.7.11 via your requirements.txt.
I made sure the versions are at least the ones listed in the readme.

I installed openmpi via: conda install openmpi

However, it appears the module torchpack.mtpack.

I also tried to go back from torch==1.9.1 to torch==1.5, but no change.

Hope you can help me.
Thanks in advance.

$ python train.py
Extension horovod.torch has not been built: /home/pepper-jk/.conda/envs/deep_comp/lib/python3.7/site-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-37m-x86_64-linux-gnu.so not found
If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.
Warning! MPI libs are missing, but python applications are still avaiable.
Traceback (most recent call last):
  File "train.py", line 15, in <module>
    from torchpack.mtpack.utils.config import Config, configs
ModuleNotFoundError: No module named 'torchpack.mtpack'

p.s. I will try this again tomorrow and update this issue if I find a solution.