Get list of models about comet HOT 5 CLOSED

On a related question, how does this dictionary map to the list of models in https://unbabel.github.io/COMET/html/models.html#faq

{'emnlp20-comet-rank': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/emnlp20-comet-rank.tar.gz',
 'wmt20-comet-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/wmt20-comet-da.tar.gz',
 'wmt20-comet-qe-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt20/wmt20-comet-qe-da.tar.gz',
 'wmt21-comet-mqm': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-mqm.tar.gz',
 'wmt21-cometinho-da': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-cometinho-da.tar.gz',
 'wmt21-comet-qe-mqm': 'https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-comet-qe-mqm.tar.gz'
}

from comet.

@alvations that's something I have to fix a long time ago already! Documentation is extremely outdated...

All other models are from our 2021 paper and are not in that documentation!

from comet.

Regarding @mjpost request, it does print the available metrics when you pass an invalid model name:

comet-score -s src.de -t hyp1.en -r ref.en --gpus 0 --model test

usage: comet-score [-h] [-s SOURCES] [-t TRANSLATIONS [TRANSLATIONS ...]] [-r REFERENCES] [-d SACREBLEU_DATASET] [--batch_size BATCH_SIZE] [--gpus GPUS] [--accelerator {dp,ddp}] [--to_json TO_JSON]
                   [--model MODEL] [--model_storage_path MODEL_STORAGE_PATH] [--mc_dropout MC_DROPOUT] [--seed_everything SEED_EVERYTHING] [--num_workers NUM_WORKERS] [--disable_bar] [--disable_cache]
                   [--disable_length_batching] [--print_cache_info]
comet-score: error: test is not a valid checkpoint path or model choice. Choose from dict_keys(['emnlp20-comet-rank', 'wmt20-comet-da', 'wmt20-comet-qe-da', 'wmt21-comet-mqm', 'wmt21-cometinho-da', 'wmt21-comet-qe-mqm'])

from comet.

When I pass --model wmt21-comet-qe-da

Global seed set to 12
Downloading refless-wmt21-comet-da.tar.gz
refless-wmt21-comet-da.tar.gz: 0.00B [00:00, ?B/s]
Traceback (most recent call last):
File "comet/cli/score.py", line 327, in
score_command()
File "comet/cli/score.py", line 170, in score_command
model_path = download_model(cfg.model, saving_directory=cfg.model_storage_path)
File "/content/COMET/comet/download_utils.py", line 209, in download_model
available_metrics[model], directory=saving_directory
File "/content/COMET/comet/download_utils.py", line 165, in download_file_maybe_extract
urllib.request.urlretrieve(url, filename=filepath, reporthook=_reporthook(t))
File "/usr/lib/python3.7/urllib/request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

which means the model name (taken from doc) is not wrong according to your post, but not able to download.
I'll use wmt20-comet-qe-da instead

from comet.

you are right @vince62s! The link is broken for that model. I'll create a new issue for this

from comet.