iseekwonderful / hpa-singlecell-2nd-dual-head-pipeline Goto Github PK

4
[ √ ] Landmark!
[0, 1, 2, 3]
[ √ ] Using #0,1,2,3 GPU
['0', '1', '2', '3']
[ ! ] Full fold coverage training! for fold: 0
[ √ ] Using transformation: s_0220/sin_256_final & None, image size: 256
[ i ] The length of train_dl is 7028, valid dl is 4391
Using cache found in /home/users/allstaff/thomas.e/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master
tf_efficientnet_b3
[ i ] Model: tf_efficientnet_b3, loss_func: bce, optimizer: Adam
parallel
4
[ ! ] pos weight: 0.1
[ √ ] Basic training
  0%|          | 0/7028 [00:12<?, ?it/s]
Traceback (most recent call last):
  File "main.py", line 172, in <module>
    basic_train(cfg, model, train_dl, valid_dl, loss_func, optimizer, result_path, scheduler, writer)
  File "/vast/scratch/users/thomas.e/HPA-singlecell-2nd-dual-head-pipeline/basic_train.py", line 93, in basic_train
    cell, exp = model(ipt, cfg.experiment.count)
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 161, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 171, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/stornext/HPCScratch/home/thomas.e/.conda/envs/rxrx/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/vast/scratch/users/thomas.e/HPA-singlecell-2nd-dual-head-pipeline/models/efficient.py", line 64, in forward
    viewed_pooled = pooled.view(-1, cnt, pooled.shape[-1])
RuntimeError: shape '[-1, 16, 1536]' is invalid for input of size 36864