How to get vocab.yml file when doing train->eval->dist about opus-mt-train HOT 6 CLOSED

Nevermind, I saw someone else's comment about marian-vocab. It might be useful to add it to the documentation somewhere? Thanks!

from opus-mt-train.

Actually I'm going to reopen this as I'm still confused - I successfully created a vocab.yml file by concatenating the source and target vocabs and passing to marian-vocab, but when I try to convert my packaged model to Huggingface I get an error: Original vocab size {opus_state.cfg['vocab_size']} and new vocab size {len(tokenizer.encoder)} mismatched AssertionError: Original vocab size 32001 and new vocab size 61724 mismatched. Is there something I'm missing here? Should I ask this question over at Huggingface?

from opus-mt-train.

Looked at the commits and figured out that the old yml vocab can be produced if using USE_SPM_VOCAB=0 flag when creating the data. Does that mean that for now, I should train all models with that flag if I want to port them over to Huggingface?

from opus-mt-train.

Yes, sorry, this is what you need now but I will talk to the people at huggingface to also support the plain text vocab files that are taken from the sentence piece models. It's a bit of a moving target.

from opus-mt-train.

Great, thanks - happy to follow up with them or help with the porting if I can. Feel free to leave this issue up or close it if you think it's not directly relevant for this project, up to you.

from opus-mt-train.

Changed it now to have USE_SPM_VOCAB=0 as default. Seems more backward compatible with everything ...

from opus-mt-train.