Separate òwe corpus into EN & YO about yoruba-text HOT 2 CLOSED

Thinking about this, would a CSV not be a better representation of the data than a text file? We can have a csv file with the format:
|yoruba|english| translation

I suppose the challenge with that would be updating the file when we get more data. Would be easier to dump into a folder than having to update a csv each time.

I'll go ahead and implement your idea now. If the csv idea appeals enough, we can always have that implementation available later.

from yoruba-text.

I think the problem with a CSV |yoruba|english| translation is not that its not a good idea or intuitive format ... it is that that is not the format used by OpenNMT, Tensor2Tensor etc.

See here (which is also how we do it for ADR), how they split text into "source" (langA) and target (langB) files ...each line corresponding to the same line in the other file: http://opennmt.net/OpenNMT-py/options/preprocess.html

So if you use a CSV, you still have to segment it into sources & targets before you can create an input-training batch, that's why I was thinking if we do any work to split it, why not make it ready for either a monolingual YO ADR task as well as a NMT task ...any sha, that's my thinking.

Back to CSV, another idea, is to ALSO make a CSV that can be used for human validation of the text or another non-NMT, non-ADR purpose.

For maintenance/update, I haven't found a good way update text, since we get text submissions periodically. For Kọ́lá's and Toluwase's texts, what I did was concatenate all texts (new & old) then did (on Linux) uniq | sort to filter out the duplicates and sort them (nice but unnecessary for training).

Let's feel free to establish our own "best practise" for our specific computing tasks!

from yoruba-text.