johannbrehmer / manifold-flow Goto Github PK

Hi, excellent work here.
I encountered NaN error when training with the config configs/train_mf_gan64d_april.config:
Traceback (most recent call last): File "/home/urkax/project/GenFed/manifold-flow-public/experiments/train.py", line 592, in <module> learning_curves = train_model(args, dataset, model, simulator) File "/home/urkax/project/GenFed/manifold-flow-public/experiments/train.py", line 504, in train_model learning_curves = train_manifold_flow_sequential(args, dataset, model, simulator) File "/home/urkax/project/GenFed/manifold-flow-public/experiments/train.py", line 276, in train_manifold_flow_sequential learning_curves = trainer1.train( File "/home/urkax/project/GenFed/manifold-flow-public/experiments/training/trainer.py", line 307, in train loss_train, loss_val, loss_contributions_train, loss_contributions_val = self.epoch( File "/home/urkax/project/GenFed/manifold-flow-public/experiments/training/trainer.py", line 380, in epoch batch_loss, batch_loss_contributions = self.batch_train( File "/home/urkax/project/GenFed/manifold-flow-public/experiments/training/trainer.py", line 513, in batch_train loss_contributions = self.forward_pass(batch_data, loss_functions, forward_kwargs=forward_kwargs, custom_kwargs=custom_kwargs) File "/home/urkax/project/GenFed/manifold-flow-public/experiments/training/trainer.py", line 633, in forward_pass self._check_for_nans("Reconstructed data", x_reco) File "/home/urkax/project/GenFed/manifold-flow-public/experiments/training/trainer.py", line 122, in _check_for_nans raise NanException training.trainer.NanException

I am using 5 GPUs, pytorch 1.7.1
Have you ever encountered such problem?

The current size of the repo is ~400 MB. I think it would be a good idea to either quash commits or create a separate public branch which isn't bloated.

Thanks a lot for releasing the code. :)

How can I get the dataset needed for the experiment?

When I call the function, it shows the error that "No such file"

Hi!

It seems there's a problem with the alternating training of the code on repo.
Upon trying to train using this option, you'd get an error saying:

Traceback (most recent call last):
  File "C:/M-Flows/experiments/train.py", line 503, in <module>
    learning_curves = train_model(args, dataset, model, simulator)
  File "C:/M-Flows/experiments/train.py", line 433, in train_model
    learning_curves = train_manifold_flow_alternating(args, dataset, model, simulator)
  File "C:/M-Flows/experiments/train.py", line 230, in train_manifold_flow_alternating
    **meta_kwargs,
  File "C:\M-Flows\experiments\training\alternate.py", line 159, in train
    **trainer_kwargs_
  File "C:\M-Flows\experiments\training\trainer.py", line 456, in partial_epoch
    batch_data, loss_functions, loss_weights, optimizer, clip_gradient, forward_kwargs=forward_kwargs, custom_kwargs=custom_kwargs
TypeError: batch_train() missing 1 required positional argument: 'parameters'

I tried to get rid of this problem by adding the parameters in AlternateTrainer to the partial_epoch, but it raises another error.
Could you please tell me how should I correct this?

Thanks.