Getting machine-translated prompts of xP3mt about xmtf HOT 4 CLOSED

Is there a way I can get the machine-translated prompts per task? / For example, how would I get the Spanish (es) prompt for Paws-x only?

The prompts are here: https://github.com/Muennighoff/promptsource/blob/xp3mt/promptsource/templates/paws-x/es/templates.yaml

Is there a way I can get the input, output pairs for Paws-x only?

You can just download the paws-x files: https://huggingface.co/datasets/bigscience/xP3mt/tree/main/es e.g. https://huggingface.co/datasets/bigscience/xP3mt/blob/main/es/xp3_paws-x_es_train_task_description-no-label_esmt.jsonl

Also see the usage guidelines here that may help: https://huggingface.co/datasets/Muennighoff/xP3x#usage

Also, more generally, how do you do machine-translation for prompts if the language is from right-to-left instead of left-to-right or has different ordering like subject-object-verb instead of subject-verb-object? Would the target come before the input or would you reorder the sentences in the input (i.e premise or hypothesis) in the prompt? And if the target comes before the input, how would the model work since it generates from left to right?

We use Google Machine Translate to translate the prompts and then just put them in the same place for all languages. For right-to-left languages like Arabic everything is the same (i.e. they are processed from beginning of sentence to the end). Usually browsers handle displaying it as right-to-left so we can treat it as left-to-right in the modelling phase.

from xmtf.

Thank you for the quick response and the pointer. It is very helpful.

In the templates https://github.com/Muennighoff/promptsource/blob/xp3mt/promptsource/templates/paws-x/es/templates.yaml, I saw the metrics was Null, but it seems they have answer choices and the original prompt https://github.com/Muennighoff/promptsource/blob/xp3mt/promptsource/templates/paws/labeled_final/templates.yaml has Accuracy as a metric? Could I still use Accuracy as a metric for the machine-translated prompts?

from xmtf.

Yes you can use accuracy. The metric field in that file is never used.

from xmtf.

Great. Thank you for all your help!

from xmtf.