Comments (14)
@MaxMax2016 你好,我做了个对比的实验,使用不同ppg训练声学模型的情况。训练集的loss如下:
whisper:medium
contentvec: checkpoint_best_legacy_500
hubertsoft: hubert-soft-0d54a1f4.pt
from lora-svc.
@hongwen-sun 赞,有疑惑然后去验证,是个搞科研的好手!
from lora-svc.
@MaxMax2016 您好,可以加一下微信吗,可以做一些交流?
[email protected] 如果可以的话,可以发邮件给我,我加您
from lora-svc.
Btw, there is argumentation behind this that whisper was actually trained on songs. It is very good at speech recognition of songs compared to many other speech models.
from lora-svc.
whisper is from openai .
from lora-svc.
mm.. does that mean it's better?
from lora-svc.
from now on, whisper is the best audio encoder for svc.这个项目的目的就是为了去证明这一点。
from lora-svc.
from now on, whisper is the best audio encoder for svc.这个项目的目的就是为了去证明这一点。
您有对比几版相同模型不同content提取方式的结果吗?比如hubert,whisper,content vec,或者类似so vit svc的第九层的content vec就用您现在的方案。
另外我去看了so vit svc的方案,用了vits的框架。相对来说,您的方案只用了声码器部分结构,相对简洁但是模型参数量和模型能力是足够的吗?
我粗浅理解是从效果上看,您的方案很难达到一个完整vits达到的效果。
from lora-svc.
@hongwen-sun 你我的目标不同,您期望的是一个音质效果好的SVC;我是在做技术方案的研究,比如whisper用于歌声转换的有效性,还有说话人自适应;我需要一个高效的方案来验证我的一些想法。
from lora-svc.
此外,我个人比较崇尚简洁。您如何看待 ‘奥卡姆剃刀原理’ 呢?
from lora-svc.
我感觉简洁也可以达到比较好的效果,并不冲突,效果的差距通过block结构的简单修改和参数量的增加完全可以弥补。
另外,如果验证有效性,我理解还是需要对比不同content的提取方法在你这套框架下的效果,不然很难证明whisper是best encoder,只能说是一个可行的方案,这个基本上你实验前就知道了。
当然,您这个仓库确实是个很好的项目,我也有参考到,非常感谢。
from lora-svc.
说白了,我本来是想白嫖您的结论,看看如果您做了对比,哪个好一点
from lora-svc.
whisper是目前为止能获取到的效果最好的多语言ASR模型,它的好坏由他的识别率就可以直接体现,它通过海量的多语言数据训练得到,这是其他开源模型无法媲美的。如果您要做语音转换可能hubert等其他自监督模型比较合适,它们包含了其他非语言的信息、如语气&情感等,也许还有性别信息、以及可能的泄漏的音色;
from lora-svc.
好的,非常感谢。
对于转换而言,whisper也不一定就比不过自监督模型,更可能的情况是这些模型都有自身的缺点,需要针对vc任务有专门的设计。
from lora-svc.
Related Issues (20)
- 请问48K比特率是否是24bit? HOT 20
- 杂音很重 HOT 10
- 换了台 Tesla P40 来跑,结果出错了 HOT 1
- Training sample size HOT 1
- Diff-svc, so-vits-svc效果对比 HOT 6
- 训练一切正常,但是保存第一个模型时出现了ZeroDivisionError: float division by zero HOT 2
- 音色是否可以融合或者调整 HOT 1
- `svc_preprocess_f0.py` rootPath should be changed to 48k? HOT 1
- ValueError: not enough values to unpack (expected 4, got 1) HOT 4
- I have a question about using lora for fine-tuning HOT 4
- you implementing vocos? HOT 3
- incorrect audio shape HOT 1
- num_samples=0 HOT 2
- .
- DataSet
- data_raw
- audio examples HOT 1
- Training is ineffective, the generator's loss is only around 5,000
- Confused HOT 1
- whisper-v3 got open sourced
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lora-svc.