torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB (GPU 0; 23.99 GiB total capacity; 7.31 GiB already allocated; 14.07 GiB free; 7.36 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I run it in a server with Tesla M40.
In loadding model,it's show

Getting T5 embs...
Traceback (most recent call last):
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/routes.py", line 437, in run_predict
output = await app.get_blocks().process_api(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/blocks.py", line 1352, in process_api
result = await self.call_function(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/blocks.py", line 1077, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/gradio/helpers.py", line 602, in tracked_fn
response = fn(*args)
File "/media/maoweicao/DATA1/IF-webui/webui.py", line 208, in generate_images
result = dream(
File "/media/maoweicao/DATA1/IF-webui/deepfloyd_if/pipelines/dream.py", line 64, in dream
t5_embs = t5.get_text_embeddings(prompt)
File "/media/maoweicao/DATA1/IF-webui/deepfloyd_if/modules/t5.py", line 96, in get_text_embeddings
text_encoder_embs = self.model(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward
output = old_forward(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1846, in forward
encoder_outputs = self.encoder(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1040, in forward
layer_outputs = layer_module(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward
output = old_forward(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 673, in forward
self_attention_outputs = self.layer[0](
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward
output = old_forward(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 579, in forward
attention_output = self.SelfAttention(
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/accelerate/hooks.py", line 156, in new_forward
output = old_forward(*args, **kwargs)
File "/home/maoweicao/miniconda3/envs/if-webui/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 509, in forward
scores = torch.matmul(
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmStridedBatchedExFix(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)

so this model only running in with support FP16 compute card?

How long exactly does it take to generate an image ?
Cause when I run it, it just keeps running, doesn't generate anything just keeps increasing STEP count but nothing happens.
Also my VRAM consumption only gets upto 6.8GB while I have 3090 so it should take advantage of all 24GB available.

Any fix to this ? Or do I need to wait longer ? I have waited 10 min, but nothing happens.

Hey! Is there a chance we could get LoRA loading as described here huggingface/diffusers#3360 in the webui?

A conda environment to run the web UI is attached.
Download : DeepFloyd_IF.zip

or copy paste this to yaml file to import to anaconda :

name: DeepFloyd_IF
channels:

• pytorch
• nvidia
• conda-forge
• defaults
  dependencies:
• abseil-cpp=20211102.0=h0e60522_0
• accelerate=0.15.0=pyhd8ed1ab_0
• aiofiles=23.1.0=pyhd8ed1ab_1
• aiohttp=3.8.1=py310he2412df_1
• aiosignal=1.3.1=pyhd8ed1ab_0
• altair=5.0.1=pyhd8ed1ab_1
• antlr-python-runtime=4.9.3=pyhd8ed1ab_1
• anyio=3.7.1=pyhd8ed1ab_0
• arrow-cpp=11.0.0=py310h7cee713_0
• asttokens=2.0.5=pyhd3eb1b0_0
• async-timeout=4.0.2=pyhd8ed1ab_0
• attrs=23.1.0=pyh71513ae_1
• aws-c-common=0.4.57=ha925a31_1
• aws-c-event-stream=0.1.6=hd77b12b_5
• aws-checksums=0.1.9=ha925a31_0
• aws-sdk-cpp=1.8.185=hd77b12b_0
• backcall=0.2.0=pyhd3eb1b0_0
• beautifulsoup4=4.11.1=py310haa95532_0
• blas=1.0=mkl
• boost-cpp=1.78.0=h5b4e17d_0
• bottleneck=1.3.5=py310h9128911_0
• brotli=1.0.9=h2bbff1b_7
• brotli-bin=1.0.9=h2bbff1b_7
• brotlipy=0.7.0=py310h2bbff1b_1002
• bzip2=1.0.8=he774522_0
• c-ares=1.19.0=h2bbff1b_0
• ca-certificates=2023.7.22=h56e8100_0
• certifi=2023.7.22=pyhd8ed1ab_0
• cffi=1.15.1=py310h2bbff1b_3
• charset-normalizer=2.0.4=pyhd3eb1b0_0
• click=8.1.6=win_pyh7428d3b_0
• colorama=0.4.6=py310haa95532_0
• comm=0.1.2=py310haa95532_0
• contourpy=1.0.5=py310h59b6b97_0
• cryptography=41.0.2=py310h31511bf_0
• cuda-cccl=12.2.128=0
• cuda-cudart=11.8.89=0
• cuda-cudart-dev=11.8.89=0
• cuda-cupti=11.8.87=0
• cuda-libraries=11.8.0=0
• cuda-libraries-dev=11.8.0=0
• cuda-nvrtc=11.8.89=0
• cuda-nvrtc-dev=11.8.89=0
• cuda-nvtx=11.8.86=0
• cuda-profiler-api=12.2.128=0
• cuda-runtime=11.8.0=0
• cycler=0.11.0=pyhd3eb1b0_0
• dataclasses=0.8=pyhc8e2a94_3
• datasets=2.14.3=pyhd8ed1ab_0
• debugpy=1.6.7=py310hd77b12b_0
• decorator=5.1.1=pyhd3eb1b0_0
• dill=0.3.7=pyhd8ed1ab_0
• exceptiongroup=1.1.2=pyhd8ed1ab_0
• executing=0.8.3=pyhd3eb1b0_0
• fastapi=0.101.0=pyhd8ed1ab_0
• ffmpeg=4.3.1=ha925a31_0
• ffmpy=0.3.0=pyhb6f538c_0
• filelock=3.9.0=py310haa95532_0
• fonttools=4.25.0=pyhd3eb1b0_0
• freetype=2.12.1=ha860e81_0
• frozenlist=1.3.3=py310h2bbff1b_0
• fsspec=2023.4.0=py310haa95532_0
• ftfy=6.1.1=pyhd8ed1ab_0
• gflags=2.2.2=ha925a31_1004
• giflib=5.2.1=h8cc25b3_3
• glib=2.69.1=h5dc1a3c_2
• glog=0.5.0=h4797de2_0
• gradio=3.23.0=pyhd8ed1ab_0
• gst-plugins-base=1.18.5=h9e645db_0
• gstreamer=1.18.5=hd78058f_0
• h11=0.14.0=pyhd8ed1ab_0
• h2=4.1.0=pyhd8ed1ab_0
• hpack=4.0.0=pyh9f0ad1d_0
• httpcore=0.17.3=pyhd8ed1ab_0
• httpx=0.24.1=pyhd8ed1ab_0
• huggingface_hub=0.15.1=py310haa95532_0
• hyperframe=6.0.1=pyhd8ed1ab_0
• icu=58.2=ha925a31_3
• idna=3.4=py310haa95532_0
• importlib-metadata=6.8.0=pyha770c72_0
• importlib_metadata=6.8.0=hd8ed1ab_0
• importlib_resources=6.0.0=pyhd8ed1ab_1
• intel-openmp=2023.1.0=h59b6b97_46319
• ipykernel=6.25.0=py310h9909e9c_0
• ipython=8.12.0=py310haa95532_0
• ipywidgets=8.0.4=py310haa95532_0
• jedi=0.18.1=py310haa95532_1
• jinja2=3.1.2=pyhd8ed1ab_1
• joblib=1.3.0=pyhd8ed1ab_1
• jpeg=9e=h2bbff1b_1
• jsonschema=4.17.3=pyhd8ed1ab_0
• jupyter_client=8.1.0=py310haa95532_0
• jupyter_core=5.3.0=py310haa95532_0
• jupyterlab_widgets=3.0.5=py310haa95532_0
• kiwisolver=1.4.4=py310hd77b12b_0
• krb5=1.19.4=h5b6d351_0
• lerc=3.0=hd77b12b_0
• libbrotlicommon=1.0.9=h2bbff1b_7
• libbrotlidec=1.0.9=h2bbff1b_7
• libbrotlienc=1.0.9=h2bbff1b_7
• libclang=14.0.6=default_hb5a9fac_1
• libclang13=14.0.6=default_h8e68704_1
• libcublas=11.11.3.6=0
• libcublas-dev=11.11.3.6=0
• libcufft=10.9.0.58=0
• libcufft-dev=10.9.0.58=0
• libcurand=10.3.3.129=0
• libcurand-dev=10.3.3.129=0
• libcurl=8.1.1=h86230a5_2
• libcusolver=11.4.1.48=0
• libcusolver-dev=11.4.1.48=0
• libcusparse=11.7.5.86=0
• libcusparse-dev=11.7.5.86=0
• libdeflate=1.17=h2bbff1b_0
• libffi=3.4.4=hd77b12b_0
• libiconv=1.16=h2bbff1b_2
• libnpp=11.8.0.86=0
• libnpp-dev=11.8.0.86=0
• libnvjpeg=11.9.0.86=0
• libnvjpeg-dev=11.9.0.86=0
• libogg=1.3.5=h2bbff1b_1
• libpng=1.6.39=h8cc25b3_0
• libprotobuf=3.20.3=h23ce68f_0
• libsodium=1.0.18=h62dcd97_0
• libssh2=1.10.0=hcd4344a_2
• libthrift=0.15.0=h636ae23_0
• libtiff=4.5.0=h6c2663c_2
• libuv=1.44.2=h8ffe710_0
• libvorbis=1.3.7=he774522_0
• libwebp=1.2.4=hbc33d0d_1
• libwebp-base=1.2.4=h2bbff1b_1
• libxml2=2.10.3=h0ad7f3c_0
• libxslt=1.1.37=h2bbff1b_0
• linkify-it-py=2.0.0=pyhd8ed1ab_0
• lz4-c=1.9.4=h2bbff1b_0
• markdown-it-py=2.2.0=pyhd8ed1ab_0
• markupsafe=2.1.1=py310h2bbff1b_0
• matplotlib=3.7.1=py310haa95532_1
• matplotlib-base=3.7.1=py310h4ed8f06_1
• matplotlib-inline=0.1.6=py310haa95532_0
• mdit-py-plugins=0.3.3=pyhd8ed1ab_0
• mdurl=0.1.0=pyhd8ed1ab_0
• mkl=2023.1.0=h8bd8f75_46356
• mkl-service=2.4.0=py310h2bbff1b_1
• mkl_fft=1.3.6=py310h4ed8f06_1
• mkl_random=1.2.2=py310h4ed8f06_1
• mpmath=1.3.0=pyhd8ed1ab_0
• multidict=6.0.2=py310h2bbff1b_0
• multiprocess=0.70.14=py310haa95532_0
• munkres=1.1.4=py_0
• nest-asyncio=1.5.6=py310haa95532_0
• networkx=3.1=pyhd8ed1ab_0
• ninja=1.10.2=haa95532_5
• ninja-base=1.10.2=h6d14046_5
• numexpr=2.8.4=py310h2cd9be0_1
• numpy=1.25.0=py310h055cbcc_0
• numpy-base=1.25.0=py310h65a83cf_0
• omegaconf=2.3.0=pyhd8ed1ab_0
• openssl=1.1.1v=h2bbff1b_0
• orc=1.7.4=h623e30f_1
• orjson=3.9.2=py310h48ea969_0
• packaging=23.0=py310haa95532_0
• pandas=1.5.3=py310h4ed8f06_0
• parso=0.8.3=pyhd3eb1b0_0
• pcre=8.45=hd77b12b_0
• pickleshare=0.7.5=pyhd3eb1b0_1003
• pillow=9.4.0=py310hd77b12b_0
• pip=23.2.1=py310haa95532_0
• pkgutil-resolve-name=1.3.10=pyhd8ed1ab_0
• platformdirs=2.5.2=py310haa95532_0
• ply=3.11=py310haa95532_0
• prompt-toolkit=3.0.36=py310haa95532_0
• protobuf=3.20.3=py310h5588dad_1
• psutil=5.9.0=py310h2bbff1b_0
• pure_eval=0.2.2=pyhd3eb1b0_0
• pyarrow=11.0.0=py310h790e06d_0
• pycparser=2.21=pyhd3eb1b0_0
• pydantic=1.10.8=py310h2bbff1b_0
• pydub=0.25.1=pyhd8ed1ab_0
• pygments=2.15.1=py310haa95532_1
• pyopenssl=23.2.0=py310haa95532_0
• pyparsing=3.0.9=py310haa95532_0
• pyqt=5.15.7=py310hd77b12b_0
• pyqt5-sip=12.11.0=py310hd77b12b_0
• pyrsistent=0.18.0=py310h2bbff1b_0
• pysocks=1.7.1=py310haa95532_0
• python=3.10.12=h966fe2a_0
• python-dateutil=2.8.2=pyhd3eb1b0_0
• python-multipart=0.0.6=pyhd8ed1ab_0
• python-xxhash=2.0.2=py310h2bbff1b_1
• python_abi=3.10=2_cp310
• pytorch=2.0.1=py3.10_cuda11.8_cudnn8_0
• pytorch-cuda=11.8=h24eeafa_5
• pytorch-mutex=1.0=cuda
• pytz=2023.3=pyhd8ed1ab_0
• pywin32=305=py310h2bbff1b_0
• pyyaml=6.0=py310h2bbff1b_1
• pyzmq=25.1.0=py310hd77b12b_0
• qt-main=5.15.2=he8e5bd7_8
• qt-webengine=5.15.9=hb9a9bb5_5
• qtwebkit=5.212=h2bbfb41_5
• re2=2022.04.01=h0e60522_0
• regex=2022.7.9=py310h2bbff1b_0
• requests=2.31.0=py310haa95532_0
• sacremoses=0.0.53=pyhd8ed1ab_0
• semantic_version=2.10.0=pyhd8ed1ab_0
• sentencepiece=0.1.99=py310h59b6b97_0
• setuptools=68.0.0=py310haa95532_0
• sip=6.6.2=py310hd77b12b_0
• six=1.16.0=pyhd3eb1b0_1
• snappy=1.1.9=h6c2663c_0
• sniffio=1.3.0=pyhd8ed1ab_0
• soupsieve=2.4=py310haa95532_0
• sqlite=3.41.2=h2bbff1b_0
• stack_data=0.2.0=pyhd3eb1b0_0
• starlette=0.27.0=pyhd8ed1ab_0
• sympy=1.12=pyh04b8f61_3
• tbb=2021.8.0=h59b6b97_0
• tk=8.6.12=h2bbff1b_0
• tokenizers=0.13.2=py310h49fca51_1
• toml=0.10.2=pyhd3eb1b0_0
• toolz=0.12.0=pyhd8ed1ab_0
• tornado=6.3.2=py310h2bbff1b_0
• tqdm=4.65.0=py310h9909e9c_0
• traitlets=5.7.1=py310haa95532_0
• transformers=4.28.1=pyhd8ed1ab_0
• typing-extensions=4.7.1=py310haa95532_0
• typing_extensions=4.7.1=py310haa95532_0
• tzdata=2023c=h04d1e81_0
• uc-micro-py=1.0.1=pyhd8ed1ab_0
• urllib3=1.26.16=py310haa95532_0
• utf8proc=2.6.1=h2bbff1b_0
• uvicorn=0.23.2=py310h5588dad_0
• vc=14.2=h21ff451_1
• vs2015_runtime=14.27.29016=h5e58377_2
• wcwidth=0.2.5=pyhd3eb1b0_0
• websockets=10.4=py310h2bbff1b_1
• wheel=0.38.4=py310haa95532_0
• widgetsnbextension=4.0.5=py310haa95532_0
• win_inet_pton=1.1.0=py310haa95532_0
• xxhash=0.8.0=h2bbff1b_3
• xz=5.4.2=h8cc25b3_0
• yaml=0.2.5=he774522_0
• yarl=1.7.2=py310he2412df_2
• zeromq=4.3.4=hd77b12b_0
• zipp=3.16.2=pyhd8ed1ab_0
• zlib=1.2.13=h8cc25b3_0
• zstd=1.5.5=hd43e919_0
• pip:
  • clip==1.0
  • diffusers==0.16.1
  • torchaudio==2.0.2
  • torchvision==0.15.2

Is this, or could this UI be made, compatible with this Mac-specific IF fork: https://github.com/brkirch/DeepFloyd-IF-example-Mac? The fork works but is hard to use because it requires editing a script file to change prompt, seed, etc.