[BUG] Uyghur Dataset: Total Items in train, dev, and test Significantly Less Than in validated.tsv about common-voice HOT 2 CLOSED

________________________________ From: Harikalar Kutusu (a.k.a. Bülent Özden) ***@***.***> Sent: Thursday, December 21, 2023 7:52:49 PM To: common-voice/common-voice ***@***.***> Cc: Osman Tursun ***@***.***>; Mention ***@***.***> Subject: Re: [common-voice/common-voice] [BUG] Uyghur Dataset: Total Items in train, dev, and test Significantly Less Than in validated.tsv (Issue #4277) Hey @neouyghur<https://github.com/neouyghur>, this is because you have too few sentences (text-corpus) wrt total number of recordings. The splits are created with the CorporaCreator repo and in its default settings, a sentence is only taken once, so it is the limiting factor. It is the most strict splitting algorithm, which also ensures that there is no transcription bias. On the other hand, my experiments show it fatally reduces the diversity in the training set. For me, the default splits are completely useless - except verry large datasets. There is some related issue on the CC repo: common-voice/CorporaCreator#113<common-voice/CorporaCreator#113> But you can use the CC to relax that, with the -s flag, e.g. -s 5 allows up to 5 recordings per sentence. But, anyway, you need to work your the text corpus, more sentences, more vocabulary, more phonemes are needed... Shameless self promotion: You can see detailed statistics of your dataset here: https://analyzer.cv-toolbox.web.tr/examine/ug/15.0 It also provides TSV files and statistics for some alternative splitting algorithms. Info is on the homepage of that webapp. — Reply to this email directly, view it on GitHub<#4277 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABQRQ3B6V5SI6UWTG2HLBDDYKQBHDAVCNFSM6AAAAABAY5QNRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRVHE3DKMJUGU>. You are receiving this because you were mentioned.Message ID: ***@***.***>