train.py only works under distributed settings about pi-gan HOT 3 CLOSED

Hi @xvdp,

Thanks for pointing out these issues. To clarify, were you not able to run train.py when you just specified one GPU with CUDA_VISIBLE_DEVICES?

If you'd like to make a PR to fix the dataset_path for cats and carla that'd be great! To guarantee reproducibility of results let's not merge in the linting changes. For non-trivial changes we'd need to retrain models and confirm we can get the same FID scores.

from pi-gan.

hi @marcoamonteiro
Im working on a setup that uses cuda just fine, but does not have CUDA_VISIBLE_DEVICES. Maybe I set up my environment incorrectly but so far in a couple of years and dozens of projects ran on pytorch, tensorflow and jax, this is the only one where I found I need to set it up. I suppose I should set up a docker with your configurations and test it but that deviates from tasks at hand.

FYI: There are a few other issues in there I noticed - in most cases loading and saving needs to be done by state_dict(), not directly saving and loading the module. I'm not sure how you are avoiding that.
Also you don't create os.path.join(opt.output_dir, 'evaluation/generated')) folder anywhere in the train code. You have a ton of double class definitions. You request a device which you never pass in def create_world2cam_matrix(forward_vector, origin)

Im of the opinion that the more general setup is the one you want out there, and yet I get it that you dont want to change things that worked for your paper, so never mind .

Having worked with various nerfs, siren and stylegans, I read thru the code and I like what you are doing, it is logical. Having tried modifying and combining those papers myself however I am curious at the thought process - the failure points. You arrived at a great point, but where did you find dead ends. Forgive my stream of thought, I know this is no forum. In any case, good work.

from pi-gan.

And yes, you are right, the fix to multigpu is much cleaner by putting, insinde of __main__

    os.makedirs(opt.output_dir, exist_ok=True)
    os.makedirs(osp.join(opt.output_dir, 'evaluation/generated'), exist_ok=True)
    
    if "CUDA_VISIBLE_DEVICES" not in os.environ:
        os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
        os.environ["CUDA_VISIBLE_DEVICES"]="0"
    num_gpus = len(os.environ['CUDA_VISIBLE_DEVICES'].split(','))
    mp.spawn(train, args=(num_gpus, opt), nprocs=num_gpus, join=True)

from pi-gan.