Comments (17)
This is a very interesting issue. During training the guidance scale is 1, and for some reason when the input is small it fails to generalize to higher guidance scale. I think probably during training we may have to vary the guidance scale randomly from 1 to 2 then? I will try to do this just for the 2nd stage and see if the problem disappears.
from styletts2.
@AWAS666 It still works even with a single word
wink.
. It did generate noise if there is no punctuation after this. I think this is caused by the training data again, where all sentences end with some sort of punctuation. https://vocaroo.com/1728QKrk6PSU https://vocaroo.com/1iptqIqXNtRj
You are correct, adding the fullstop helps fix it.
But only if you have embedding scale at 1, as soon as you raise that, it does it again :)
from styletts2.
Nvm also seems to happen on single words, setting both alpha and beta to zero makes it return to normal though.
from styletts2.
This is not supposed to occur. Setting alpha and beta to 0 means not using the diffusion model at all. What are your packages versions?
from styletts2.
Windows 10
Python 3.11.4
Torch 2.1.0+cu118
And loading the model to the GPU instead of CPU, but I'll do some further testing to narrow it down
This doesn't seem to make a difference...
from styletts2.
Could you make a conda environment with Python 3.10 instead? You can run the colab demo and check package versions there and make sure you install these packages instead.
from styletts2.
Could you make a conda environment with Python 3.10 instead? You can run the colab demo and check package versions there and make sure you install these packages instead.
I tried 3.10 locally, but exactly the same issue.
from styletts2.
Tried it in collab aswell, if I put in just the word "wink" as inference text, it will give me bad white noise.
from styletts2.
I think it could be due to not such a training sample during training. The model has never seen a single word during training (because we removed speech shorter than one second).
from styletts2.
Is there a way around it without retraining it, like dropping the diffusion model on short inputs?
Otherwise that likely means having to retrain, right?
from styletts2.
You can add some filler words before or after the word you want to speak and cut the audio to only get the word you are interested in.
from styletts2.
not a pretty solution either :)
but at least it isnt my setup alone
from styletts2.
I got the same problem - it's probably not related to the sentence length but on some first words in the sentance like: "you", "me". For example 'You can do that too." [58] - produces white noise on Colab.
So for example if in the longer text there is a sentance starting with "You will need .." this will corrupt audio afterwards.
But as mentioned alpha = 0, beta = 0
fixes that.
from styletts2.
@easyrider I have tried this You can do that too.
on Colab and was able to synthesize the speech in any voice.
https://vocaroo.com/1f8Rpq84L8H4
https://vocaroo.com/110LoHbYIP9Y
https://vocaroo.com/155vjtpiSYLO
https://vocaroo.com/19lqIdQEM9uJ (LJSpeech)
from styletts2.
@AWAS666 It still works even with a single word wink.
. It did generate noise if there is no punctuation after this. I think this is caused by the training data again, where all sentences end with some sort of punctuation.
https://vocaroo.com/1728QKrk6PSU
https://vocaroo.com/1iptqIqXNtRj
from styletts2.
Great to hear
from styletts2.
I have the same problem specifically for short sentences/phrases (all with puncutation) running on MacOS M2. I noticed that it seems to be more likely when the sentence length is less than about 40 characters. I was already doing TTS on longform audio, so I wrote a script that splits up sentences but if any sentence is less than 40 characters it attaches it to the previous or next sentence. This way every block of text I processed with StyleTTS2 is longer than 40 characters. That fixed the problem entirely for me. I didn't make any changes to punctuation and didn't change any of the words in the text.
from styletts2.
Related Issues (20)
- Getting CUDA Out of memory error in Stage2 training HOT 13
- Multi-lingual training HOT 27
- In training Stage1 after 49th epoch getting RuntimeError: you can only change requires_grad flags of leaf variables, g_loss.requires_grad = True HOT 1
- First stage training after 49th epoch (i.e., when epoch >= TMA_epoch)
- Getting error in d_loss.backward() of first_stage training
- Can the model learn accents not supported by espeak-ng?
- Joint training is failing with Assertion error HOT 2
- In 2nd stage training AttributeError: 'AudioDiffusionConditional' object has no attribute 'module'
- Questions about Differentiable Duration Modeling HOT 1
- weird chinese pronunciation HOT 3
- Training PL-BERT on styletts2-community/multilingual-pl-bert
- Can anyone please share checkpoints that we get after we complete both stages of training HOT 4
- Model Size of fine tuned Model
- Can StyleTTS2 use phonemization from different languages to finetune or train?
- StyleTTS Python API doesn't detect devanagari script
- After training 1 epoch, train_first.py crashes: RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 1, 1, 800] HOT 1
- Do we need lr scheduler? HOT 2
- ImportError: A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.1 HOT 1
- Wav File not being read
- What is the chinese phonemizer for pretrained multilinugual PL-BERT?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from styletts2.