Cuda Error with low_mem about real-time-voice-cloning HOT 16 CLOSED

Hmm, I don't have a solution right now. I'll have to look into it.

from real-time-voice-cloning.

I've found out why and how to fix it: multiprocess uses forked workers by default, which inherits some state CUDA isn't expected. Switching to spawned workers fixes it.

diff --git a/synthesizer/inference.py b/synthesizer/inference.py
index 99fb778..b9cc9c0 100644
--- a/synthesizer/inference.py
+++ b/synthesizer/inference.py
@@ -2,12 +2,12 @@ from synthesizer.tacotron2 import Tacotron2
 from synthesizer.hparams import hparams
 from multiprocess.pool import Pool  # You're free to use either one
 #from multiprocessing import Pool   # 
+from multiprocess.context import SpawnContext
 from synthesizer import audio
 from pathlib import Path
 from typing import Union, List
 import tensorflow as tf
 import numpy as np
-import numba.cuda
 import librosa
 
 
@@ -80,13 +80,15 @@ class Synthesizer:
             # Low memory inference mode: load the model upon every request. The model has to be 
             # loaded in a separate process to be able to release GPU memory (a simple workaround 
             # to tensorflow's intricacies)
-            specs, alignments = Pool(1).starmap(Synthesizer._one_shot_synthesize_spectrograms, 
-                                                [(self.checkpoint_fpath, embeddings, texts)])[0]
+            specs, alignments = Pool(1, context=SpawnContext()
+                                    ).starmap(Synthesizer._one_shot_synthesize_spectrograms, 
+                                              [(self.checkpoint_fpath, embeddings, texts)])[0]
     
         return (specs, alignments) if return_alignments else specs
 
     @staticmethod
     def _one_shot_synthesize_spectrograms(checkpoint_fpath, embeddings, texts):
+        import numba.cuda
         # Load the model and forward the inputs
         tf.reset_default_graph()
         model = Tacotron2(checkpoint_fpath, hparams)

from real-time-voice-cloning.

What does demo_cli.py give?

from real-time-voice-cloning.

Without the low_mem flag, it says that all tests pass. With the low_mem flag, I get the same CUDA_ERROR_NOT_INITIALIZED.

from real-time-voice-cloning.

Same problem.

"demo_cli.py" output without "--low_mem" flag:
Found 1 GPUs available. Using GPU 0 (GeForce MX150) of compute capability 6.1 with 2.1Gb total memory.
...
Use standard file APIs to check for files with this prefix. Testing the vocoder... All test passed! You can now synthesize speech.

"demo_cli.py" output with "--low_mem" flag:

Use standard file APIs to check for files with this prefix. E0707 15:55:23.503854 139813249828608 driver.py:321] Call to cuInit results in CUDA_ERROR_NOT_INITIALIZED
...
multiprocess.pool.RemoteTraceback: /python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 233, in initialize raise CudaSupportError("Error at driver init: \n%s:" % e) numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: [3] Call to cuInit results in CUDA_ERROR_NOT_INITIALIZED: multiprocess/pool.py", line 644, in get raise self._value numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: [3] Call to cuInit results in CUDA_ERROR_NOT_INITIALIZED:

from real-time-voice-cloning.

I have the same error

from real-time-voice-cloning.

Can you try pulling from this new branch and see what gives?

from real-time-voice-cloning.

New branch code without "--low_mem" flag get stucked on "Interactive generation loop" after "all test passed".
Output:

W0710 12:49:35.993566 140170637424384 deprecation.py:323] From /Real-Time-Voice-Cloning/env/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Testing the vocoder...
All test passed! You can now synthesize speech.
This is a GUI-less example of interface to SV2TTS. The purpose of this script is to show how you can interface this project easily with your own. See the source code for an explanation of what is happening.
Interactive generation loop

With "--low_mem" flag same cuda init error.
Output:
https://pastebin.com/PFBEbEnz

from real-time-voice-cloning.

I still run into the same issue as well with the --low_mem flag.

from real-time-voice-cloning.

When you get stucked on "Interactive generation loop", you should input your own audio file, then the code will work, you can read the source code . But I don't know why the message didn't work.

from real-time-voice-cloning.

I have the same error with 2.1G GPU!

from real-time-voice-cloning.

I can't reproduce this unfortunately, I'm running a win10 install on two computers (one with 2gb VRAM) and tensorflow-gpu=1.13.1. I'm going to need more information to look into it.

from real-time-voice-cloning.

I am running Linux on both computers. Ubuntu 18.04 with nvidia-430 and Cuda 10.2 (2gb VRAM) and Linux Mint 19.1 with nvidia-410 and Cuda 10.0 (4gb VRAM) on the other.

I have been playing around a bit with the code and it seems that in vocoder/inference.py (almost) any call to numba.cuda results in the error.

from real-time-voice-cloning.

I've implemented it in a rush on the low_mem_fix branch, can you try and see what gives?

from real-time-voice-cloning.

demo_cli.py now passes but I've run into another bug with demo_toolbox.py so I can't verify that one.

from real-time-voice-cloning.

I've got the same problem and can confirm that @lilydjwg's solution solved it.

from real-time-voice-cloning.