Causal model running on GPU about performer-pytorch HOT 7 CLOSED

@Warvito I'm in the dark as much as you are :( I have been putting off custom CUDA code for as long as I could, but the results of this paper was irresistible

from performer-pytorch.

I had the same issue. I am not sure what worked for me but after some steps training with casual=True is working.

my steps:

1. Add CUDA to the PATH variable
   export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
2. Create LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
3. Create new environment and install fast-transformers as in that issue comment: idiap/fast-transformers#23 (comment)
4. Install performer-pytorch after that

from performer-pytorch.

@Warvito ahh, so not often spoken about is the fact that the auto-regressive flavor of linear attention actually incurs a pretty big memory cost (x sequence length) and requires special CUDA code to be performant (it is probably why google chose to do this in Jax)

EPFL wrote up a nice implementation, but i think it is somehow failing to be imported on your machine https://github.com/idiap/fast-transformers/blob/master/fast_transformers/causal_product/__init__.py#L12

from performer-pytorch.

@Warvito could you try python-ing into the interactive session and run

> import fast_transformers.causal_product.causal_product_cuda

and see what happens?

from performer-pytorch.

@lucidrains Thank you for the quick reply.

I tried the command that you asked and I got the following error:

Traceback (most recent call last):
  File "/home/walter/Desktop/minGPT/venv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-87c39b6d500c>", line 1, in <module>
    import fast_transformers.causal_product.causal_product_cuda
  File "/home/walter/pycharm-2020.1.1/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ModuleNotFoundError: No module named 'fast_transformers.causal_product.causal_product_cuda'

I have the 0.3.0 version installed here, and it works as expected when using casual=False.
I tried to uninstall pytorch-fast-transformers and install it again. But it did not worked.

I had the chance to try also in a system with a V100 and CUDA 11. And it worked as expected.
I also tried it on Google colab with a Tesla T4 and CUDA 10.1. And it worked as expected. Maybe it is something related with the RTX architecture? In any case, it might be a issue from pytorch-fast-transformers.

Thank you again for the quick reply, and thank you very much for all your repositories. ^^

from performer-pytorch.

@arti32lehtonen is right, make sure c++ tool chain (gcc) and cuda tool chain (nvcc) is available in your environment. If not, use export command make it visible (try "nvcc --version" after that), then reinstall the package.

from performer-pytorch.

Thx @arti32lehtonen and @yygle !
I tried your suggestions and it worked!

from performer-pytorch.