Comments (6)
Have you tried Training a Codec that is okay for music sound effects and speech?
Higher bitrate codecs could potentially be universal, since speech can be considered a subset of the wider range of audio signals. However, at the moment, I would not recommend using snac_32khz
or snac_44khz
for speech, as they were primarily trained on music. I plan to improve them by including more speech samples to balance the training set. I hope this won't affect the music quality, while it should improve speech clarity.
Note that this approach isn't unique. Conventional codecs, such as Opus, under the hood, uses one format optimized for lower-bitrate speech coding and another for general audio.
from snac.
This is what we are working on: https://docs.google.com/document/d/1nfngl0lv3XAWLQg5z-euYcE6NGGaoH7BdyoujVf7tEc/edit?usp=sharing
:)
from snac.
I tried to reconstruct an audio file, but it fails:
https://colab.research.google.com/drive/1AfRmV1sG0aLdGWM5fWNCnc4vLa6aAB_W?usp=sharing
from snac.
Hey, your audio shape is [1, T]
, but the model expects it to be [1, 1, T]
. Just add an extra dimension to the input tensor like this, and it should work perfectly:
with torch.inference_mode():
audio_hat, _, _, _, _ = model(audio.unsqueeze(1))
Can you also provide some numbers how many Tokens one second of speech or music would take?
snac_24khz
encodes audio into discrete codes at three temporal levels with rates of 12, 23, and 47 Hz, resulting in roughly 82 tokens per second. You can also find the corresponding numbers for music models: snac_32khz and snac_44khz.
from snac.
Thank you, here is my fixed Colab: https://colab.research.google.com/drive/1AfRmV1sG0aLdGWM5fWNCnc4vLa6aAB_W?usp=sharing
If you would like to train a universal codec, we could provide you with compute and data :)
I am from https://laion.ai
from snac.
Great! I will follow up by email to discuss further
from snac.
Related Issues (10)
- About the speech synthesis results HOT 3
- import error and state dict load failure
- create bibtex HOT 1
- Architectural questions (SNAC vs Vocos) HOT 2
- Few questions about the implementation HOT 2
- a snake missing?
- use of NoiseBlock and depthwise? HOT 1
- why codebook_size=4096? HOT 1
- reconstructing audio from tokens :) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snac.