v@v-System-Product-Name:~/ShallowFF$ /bin/python3 /home/v/ShallowFF/train.py
/home/v/ShallowFF/train.py:52: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
X = np.fromstring(file.read(int(95e6)), dtype=np.uint8)
training: 0%| | 0/100000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/v/ShallowFF/train.py", line 87, in
loss = model(next(train_loader))
File "/home/v/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/v/ShallowFF/alr_transformer/at.py", line 82, in forward
logits = self.net(x_inp, **kwargs)
File "/home/v/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/v/ShallowFF/alr_transformer/model.py", line 203, in forward
x = self.transformer(x)
File "/home/v/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/v/ShallowFF/alr_transformer/model.py", line 178, in forward
x = block(x) + x
File "/home/v/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/v/ShallowFF/alr_transformer/model.py", line 138, in forward
sim = sim.masked_fill(causal_mask, -torch.finfo(sim.dtype).max)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 15.69 GiB total capacity; 1.40 GiB already allocated; 492.12 MiB free; 1.43 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF