Hi,

can somebody explain how this requires ~20GB VRAM?
For 750M+450M params that seems very strange. The readme indicates that there is room for optimization but I would like to understand what the main problem is.

File "....\MARS5-TTS\./mdl\hub\Camb-ai_mars5-tts_master\inference.py", line 291, in tts
    final_audio = self.vocode(final_output).squeeze()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".....\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "....\MARS5-TTS\./mdl\hub\Camb-ai_mars5-tts_master\inference.py", line 158, in vocode
    wav_diffusion = self.vocos.decode(features, bandwidth_id=bandwidth_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......
  File "....\Lib\site-packages\torch\nn\modules\conv.py", line 306, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Calculated padded input size per channel: (6). Kernel size: (7). Kernel size can't be greater than actual input size

I not sure what is wrong i feeded a 5 second wav file and a transcript.but throws this error.

Hi, i found this project on HN, it looks interesting, but i don't want to manually setup all the things to just check the quality of your solution.

The modern delivery standard is a docker image, because it allow to any people run your code with a single command.
Check another projects from HN like https://github.com/gabotechs/MusicGPT

So please, create docker file that will encapsulate whole configuration and let users to try your solution simple and fast

safetensors is a file format for tensor storage that is faster to load than PyTorch pickles and avoids the security risks of loading pickles.

It's true that if the code and the weights come from the same repo, one is just as safe as the other. But if there's any chance that you're going to end up with an ecosystem where various different models or fine-tunes are floating around, I think it best to set precedent for using safetensors from the start.

The demo link to https://179c54d254f7.ngrok.app/ on the github page for this project is broken

Great work!!! 😮😮😮
Would be awesome to have portuguese support.
Is there anything that we can help on this?

Hello!
I want to use your tts locally in my project, but I need support for eng, russian and kazakh. Is there any documentation on these languages ​​in Mars5?

Noting We are open-sourcing MARS5 in English under GNU AGPL 3.0, but you can request to use it under a different license by emailing [[email protected]](mailto:[email protected]).

Keep in mind, if you are accepting third party contributions on github, the offering of another licenses is very dubious. You should consult a lawyer before you do so.

I have a PR for fixing the numpy requirement, and honestly I don't care (it's one line fixing a trivial bug, whatever), but others might care and it opens you to potential lawsuits from contributors by relicensing their contributions without their authorization.

Please if you could guide me how to make style transfer of any voice using this tool? and how much long text could this get transform into voice over?

Takes too long for deep clone to finish processing even for a simple text (i9+ 32gb, RTX 3080 12gb) and the result is also not that good. We need optimizations. Shallow clone does not work. The output is something else other than the text provided.
output_audio.zip

The quality is really bad when reference transcript is not provided, even in the mars5_demo notebook.

However, it works well on the official mars5 speech emulation demo website, even when the reference transcript is not provided.

Does it use tools like whisper to generate the audio transcript first?

Thanks!

The issue:

RuntimeError                              Traceback (most recent call last)

[<ipython-input-7-2d05018561f0>](https://localhost:8080/#) in <cell line: 6>()
      4                       top_k=100, temperature=0.7, freq_penalty=3)
      5 
----> 6 ar_codes, wav_out = mars5.tts("The quick brown rat.", wav, 
      7           ref_transcript,
      8           cfg=cfg)

13 frames

[~/.cache/torch/hub/Camb-ai_mars5-tts_master/mars5/nn_future.py](https://localhost:8080/#) in forward(self, x, freqs_cis, positions, mask, cache)
    249             scatter_pos = (positions[-self.sliding_window:] % self.sliding_window)[None, :, None, None]
    250             scatter_pos = scatter_pos.repeat(bsz, 1, self.n_kv_heads, self.args.head_dim)
--> 251             cache.cache_k[:bsz].scatter_(dim=1, index=scatter_pos, src=xk[:, -self.sliding_window:])
    252             cache.cache_v[:bsz].scatter_(dim=1, index=scatter_pos, src=xv[:, -self.sliding_window:])
    253 

RuntimeError: scatter(): Expected self.dtype to be equal to src.dtypeRuntimeError                              Traceback (most recent call last)

[<ipython-input-7-2d05018561f0>](https://localhost:8080/#) in <cell line: 6>()
      4                       top_k=100, temperature=0.7, freq_penalty=3)
      5 
----> 6 ar_codes, wav_out = mars5.tts("The quick brown rat.", wav, 
      7           ref_transcript,
      8           cfg=cfg)

13 frames

[~/.cache/torch/hub/Camb-ai_mars5-tts_master/mars5/nn_future.py](https://localhost:8080/#) in forward(self, x, freqs_cis, positions, mask, cache)
    249             scatter_pos = (positions[-self.sliding_window:] % self.sliding_window)[None, :, None, None]
    250             scatter_pos = scatter_pos.repeat(bsz, 1, self.n_kv_heads, self.args.head_dim)
--> 251             cache.cache_k[:bsz].scatter_(dim=1, index=scatter_pos, src=xk[:, -self.sliding_window:])
    252             cache.cache_v[:bsz].scatter_(dim=1, index=scatter_pos, src=xv[:, -self.sliding_window:])
    253 

RuntimeError: scatter(): Expected self.dtype to be equal to src.dtype

Are the weights for other languages also available to download?

When I run the Google Colab notebook, it throws an error when I reach this command, I believe it's down to a corrupt safetensors installation

mars5, config_class = torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', trust_repo=True)

I fixed it by manually upgrading safetensors and using this command

!pip install --upgrade safetensors

I'm planning to run this locally - but thought it might be useful to publish my fix and perhaps the Colab file can be updated

in case people would like to contribute adding another english accent or another languages, which documents should they refer ?

本地部署总是不成功，有没有启动包或者安装包之类的？

Thanks for the great model.
Could you briefly explain how each InferenceConfig might affect the result?

Results seem excellent, but it definitely requires too much VRAM to be run. It would be great if you could make a lighter version keeping as much quality as you can.

Thanks.

I want to fine-tune MARS5 on my own data. Do you have code to do this?

I get the following error
TypeError: _DecoratorContextManager.__call__() got an unexpected keyword argument 'cfg'

on this line

ar_codes, wav_out = mars5.tts("The quick brown rat.", wav, ref_transcript, cfg=cfg)

Installed with the requirements on Windows:

torch==2.0.1+cu117
torchvision==0.15.2+cu117
torchaudio==2.0.2+cu117
numpy==1.26.4
regex
librosa
vocos
encodec
safetensors

The output audio is not synthesized.

I see there's an official demo of Mars5 on Replicate

https://replicate.com/camb-ai/mars5-tts

Could you add the cog.yaml and predict.py files that were used to create that demo to this repo? That way users like myself who use replicate can riff on the code in a way that makes it easier to share and host demos of results.

https://github.com/AIFSH/ComfyUI-MARS5-TTS

Hi

Could you give the exact versions of dependencies in requirements.txt?

While installing on Mac, I am always getting errors because of the versions and need to google it every time to fix.
For example, numpy 2.0 was just released, but your lib does not work with it, so everyone need to use numpy==1.26, and so on.

hi, can i run tts mars5 locally with dual language support? I'm very interested in your project and would like to know more about you and mars5, I'd love some feedback or even a chat with you

By the way, you are one of the few who were able to make a decent synthesis of speech in the Kazakh language, my team and I are very surprised

I was running the quickstart demo on colab but the Synthesis step is taking too much time to run, even when I try runing it for shallow clone. Is there any possible way to fix this?

Tried to generate some outputs using this sentence from the demo's instructions:

We provide several generation candidates when you synthesize text, and attempt to pick the best one on the right.

The word "several" simply WILL NOT come out correctly. It comes out as "seeval," "seeral," "seel," etc.

I am sure this is a byproduct of being an early release, but I want to flag it now as I think that in addition to training there ought to be a way to manually pass in pronunciation data using ssml

Example:

 <phoneme alphabet="ipa" ph="ˌmænɪˈtoʊbə">manitoba</phoneme>
 <phoneme alphabet="x-sampa" ph='m@"hA:g@%ni:'>mahogany</phoneme>

This way, if the autoregressive model repeatedly guesses incorrectly (i.e. on an unusual name), there is a way to force the right result.

I feel like an idiot here. Running this on Windows (WSL with conda). Where in the heck does the audio file output get saved?

I used demo to generate this sentence 'Select a speaker, enter some text, and hit "generate" to hear Mars 5 yourself.' The number 5 is not pronounced

Will you provide the training/finetune script?

Can we load the reference voice once and then use it for multiple inferences to improve processing speed, which is currently quite slow?

It's amazing what you guys are doing! Are there any plans to add more languages, such as Arabic?

Thanks.

Hi, your project looks promising.

On the demo, in the examples of use and in the releases there is a link only to the English-language model, please tell me how you can choose another language and whether it can be chosen at all, you have stated support for 140 languages.

Or do you give only English language model within the open source code?

there is this link https://179c54d254f7.ngrok.app/

which didnt load for me

but i assume it is gradio

can you share it ? thank you

I've followed all the instructions, but on Windows I cannot get the model to load. It keeps getting stuck attempting to delete the temporary .model file and crashing.

Stacktrace:

Traceback (most recent call last):
  File "E:\Users\MyUser\Documents\git\mars5\clone_tts.py", line 6, in <module>
    mars5, config_class = torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', trust_repo=True)
  File "E:\Users\MyUser\Documents\git\mars5\.venv\lib\site-packages\torch\hub.py", line 568, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "E:\Users\MyUser\Documents\git\mars5\.venv\lib\site-packages\torch\hub.py", line 597, in _load_local
    model = entry(*args, **kwargs)
  File "E:\Users\MyUser\Documents\git\mars5\./hub\Camb-ai_mars5-tts_master\hubconf.py", line 31, in mars5_english       
    mars5 = Mars5TTS(ar_ckpt, nar_ckpt, device=device)
  File "E:\Users\Nolan\Documents\git\mars5\./hub\Camb-ai_mars5-tts_master\inference.py", line 85, in __init__
    os.remove(tfn)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmp1hrrwls1texttok.model'

The crash seems to be happening here:

When I inspect the user of this file, it's locked by python.exe, meaning the file is open somewhere in python and hasn't been closed yet.

I'll do my best to debug and add anything more, but my knowledge and expertise is not in this area.

System Info

OS: Windows 11 Pro 23H2 build 22631.3593
Processor: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz 3.79 GHz
RAM: 64.0 GB
Graphics Card: NVIDIA GeForce RTX 4090
Graphics VRAM: 24 GB
Graphics Driver: 555.99
Python Version: 3.10.11
Torch Version: 2.3.1

File "......\MARS5-TTS\inference.py", line 159, in _from_pretrained
return cls(ar_ckpt=ar_ckpt, nar_ckpt=nar_ckpt, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".....\inference.py", line 92, in init
self.texttok.load(texttok_data)
File "....\MARS5-TTS\mars5\minbpe\base.py", line 143, in load
assert model_file.endswith(".model")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

why does it have this assertion?what is the model it is finding for where can i download it?I cloned from hf repo and just copy and paste the huggingface example.

I init the enviroment with README.md guideline, when I execute the Step3:Pick a reference and optionally its transcript，get the error as flow
‘’‘
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
xxx
xxx
xxx
ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use 'numpy._import_array' to disable if you are certain you don't need it).
’‘’
I find the system default librosa version is 0.10.2.post1 and numpy version is 2.0.0.

To fix the problem, we should downgrade the librosa and numpy version

The fix method is as flow
'''
pip uninstall librosa && pip uninstall numpy
pip install librosa==0.9.1
'''

Maybe the other librosa version will work too, but 0.10.2.post1 and 0.10.2 will be not

What is the maximum duration that the model can utilize? With just 5 seconds of audio and a snippet of text

Also can you share a gradio demo file? thank you so much

Thanks for the great project! Looks like only English voice generation works well, generating audio in spanish results in a native english speaking voice talking spanish.

Also, I did testing with a spanish reference voice with similar results.

Are there plans to support other languages?

how to train the model and ft it

Hey @akshhack I'm wondering how you're dealing with licensing dual AGPL + Commercial since there are no CLAs or something, how do you incorporate external commits ? I want to do the same for my upcoming projects, your input is appreciated

Hi - just setting this up on my machine (baremetal / pyenv 3.10) and have just noticed that there is no inference script, jupyter notebook or gradio demo to verify the install?

If I can find the time I would reverse engineer what's happening here to write the script - but do you have this at hand already?

Hi,

The current notebook runs successfully on a MacBook Pro (tried on M3 Pro) but runs only on the cpu. Even if I set the torch device to mps it seems to fallback to the cpu. Is there a version / way to run this using mps for faster inferencing?

Currently, when using deep cloning, and maybe when not - the model starts producing artifacts after 12 seconds of total new audio generation. Was wondering if it is expected for current model checkpoint, or needs further troubleshooting?

I followed the QuickStart (https://github.com/Camb-ai/MARS5-TTS?tab=readme-ov-file#quickstart) and copy/pasted code exactly. (did the pip install --upgrade torch torchaudio librosa vocos encodec, etc) - and here's my script. But when I run it, it closes with error (full console log below the script):

ImportError: cannot import name 'Mars5TTS' from partially initialized module 'inference' (most likely due to a circular import) (/Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master/inference.py)

Script (mars5.py):

import torch, librosa

mars5, config_class = torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', trust_repo=True)
# The `mars5` contains the AR and NAR model, as well as inference code.
# The `config_class` contains tunable inference config settings like temperature.

# Load reference audio between 1-12 seconds.
wav, sr = librosa.load('/Users/josiahbryan/Downloads/fakePhoneConvoJosiah.wav',
                       sr=mars5.sr, mono=True)
wav = torch.from_numpy(wav)
ref_transcript = "Hey yeah okay...ummm, That sounds great. . Next, we're going to go, and, an' take it, yeah. Okay! Sounds good...when are you coming home? Perfect, okay, talk soon! Thanks, bye."

# Pick whether you want a deep or shallow clone. Set to False if you don't know prompt transcript or want fast inference. Set to True if you know transcript and want highest quality.
deep_clone = True
# Below you can tune other inference settings, like top_k, temperature, top_p, etc...
cfg = config_class(deep_clone=deep_clone, rep_penalty_window=100,
                      top_k=100, temperature=0.7, freq_penalty=3)

ar_codes, output_audio = mars5.tts("The quick brown rat.", wav,
          ref_transcript,
          cfg=cfg)
# output_audio is (T,) shape float tensor corresponding to the 24kHz output audio.

Console Output:

(base) josiahbryan@JosiahscBookPro devel % python mars5.py 
Downloading: "https://github.com/Camb-ai/mars5-tts/zipball/master" to /Users/josiahbryan/.cache/torch/hub/master.zip
Using cache found in /Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master
Traceback (most recent call last):
  File "/Users/josiahbryan/devel/rubber/backend/devel/mars5.py", line 3, in <module>
    mars5, config_class = torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', trust_repo=True)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 568, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 594, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 106, in _import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master/hubconf.py", line 7, in <module>
    from inference import Mars5TTS, InferenceConfig
  File "/Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master/inference.py", line 12, in <module>
    from mars5.model import CodecLM, ResidualTransformer
  File "/Users/josiahbryan/devel/rubber/backend/devel/mars5.py", line 3, in <module>
    mars5, config_class = torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', trust_repo=True)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 568, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 594, in _load_local
    hub_module = _import_module(MODULE_HUBCONF, hubconf_path)
  File "/Users/josiahbryan/miniforge3/lib/python3.10/site-packages/torch/hub.py", line 106, in _import_module
    spec.loader.exec_module(module)
  File "/Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master/hubconf.py", line 7, in <module>
    from inference import Mars5TTS, InferenceConfig
ImportError: cannot import name 'Mars5TTS' from partially initialized module 'inference' (most likely due to a circular import) (/Users/josiahbryan/.cache/torch/hub/Camb-ai_mars5-tts_master/inference.py)