Comments (11)
token means word piece? alphabet means character?
from wenet.
Yes, you understand correct
from wenet.
character can be used as model unit, just change the pipeline to use char. Generally, char has lower accuracy than word piece, so word piece is used as default model unit.
from wenet.
With low resource data character based model shows better results then word piece model, based on my small research.
As far as i understand, only i need to change is the format.data file for training. if i use characters to train, i have to change, for instance, "▁CHAPTER ▁ONE" to "CHAPTER ONE" and corresponding token ids and obviously tokens shape and i don't need to decode text_bpe in test stage, am i right?
from wenet.
for "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E", and the dict should be A-Z plus [space].
from wenet.
for "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E", and the dict should be A-Z plus [space].
where [space] could be also replaced with a special char "▁" to simplify data processing.
from wenet.
For dict is:
blank 0
unk 1
A 2
...
Z 28
sos/eos 29
Am i right?
from wenet.
add one special symbol for space.
from wenet.
Okey, many thanks and that's all what i need to change?
from wenet.
- the dict
- training corpus should be prepared as char, such as "CHAPTER ONE", the format.data should be "C H A P T E R [space] O N E"
from wenet.
Many thanks, sorry for dumb questions !
from wenet.
Related Issues (20)
- Triton Server - support of Unified Conformer model fails HOT 6
- MOE 模型是否可以支持导出onnx HOT 2
- Can paraformer be inferenced with gpu and runtime ? HOT 1
- NotImplemented: Subclasses of Dataset should imlement_getitem_. HOT 1
- 部署到安卓,错误率很高 HOT 1
- update torch to 2.3.0+cu121, torchaudio fail in func tar_file_and_group of wenet/dataset/datapipes.py HOT 1
- How can I set beam_size in Android runtime? HOT 2
- 在两人对话的场景中,未来有计划增加区分对话人的功能特性吗
- 使用命令时报错 HOT 1
- 训练的模型后验为空的问题 HOT 1
- 跑paraformer执行转换格式命令时报错
- paraformer模型训练报错 HOT 1
- Does the wenetruntime package still exist? HOT 2
- jit export transducer HOT 2
- Rank 1 failed to pass monitoredBarrier in 1200000 ms HOT 9
- Unified Conformer ASR模型中的attention decoder解码是否是真正的streaming? HOT 4
- 现在是否有多语种wenet模型? HOT 3
- moe模型的适配问题 HOT 1
- swbd run.sh data preparation error HOT 1
- wenet --language chinese audio.wav HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wenet.