danieldeutsch / repro Goto Github PK

Is it possible to run QAEval on cpu?
As far as I see the device param (int) is used to select which gpu to run on, but does not allow an option for cpu.

Trying to run the demo. When I run

lewis2020 = BART(model="bart.large.cnn")
summary2 = lewis2020.predict(document)

I Get
Not Found ("no such image: danieldeutsch/lewis2020:1.1: No such image: danieldeutsch/lewis2020:1.1")
and for dou
404 Client Error for http+docker://localhost/v1.41/images/danieldeutsch/dou2021:1.0/json: Not Found ("no such image: danieldeutsch/dou2021:1.0: No such image: danieldeutsch/dou2021:1.0")

But this model works
liu2019 = BertSumExtAbs(device=0)

BaryScore: https://arxiv.org/abs/2108.12463
DepthScore: https://arxiv.org/abs/2103.12711
InfoLM: https://arxiv.org/abs/2112.01589
Code repository: https://github.com/PierreColombo/nlg_eval_via_simi_measures/

Hi,

I am using qaeval. I observed a bunch of closed danieldeutsch/deutsch2021:1.0 containers on my system. Are they needed for reuse purposes or could that be avoided?

kind regards

Hi,

thanks for sharing qaeval and your work on making metrics accessible! As I think making things accessible is very important to enhance further research and community dynamics I want to share my impression on setting up qaeval metric in my pipeline.

It seems the information is scattered around different places and it might be useful to be aligned and condensed.

• How to install: advantages / disadvantages and differences between qaeval-repro, sacre-rouge and repro are not entirely clear. (qa-eval suggests sacre rouge, but with docker or not? repro doesnt suggest sacre rouge, so use it directly?)
• Differences to paper: only on qaeval-repo
• Return specification: only documented in sacre rouge, but does not match the return type in repro (also containing "lerc")(so additional differences to paper?)

https://arxiv.org/abs/1906.03221
https://github.com/ratishsp/mlb-ie
https://github.com/ratishsp/mlb-data-scripts

Hi,

has anybody else experienced issues running qaeval on recent gpus?
I am using a NVIDIA GeForce RTX 3080 Ti. As far as I see, it requires cuda >= 11 and the project requires pytorch 1.7 which is only provided with cuda up to 10.2. If I try to run the project I get an error:
RuntimeError: CUDA error: no kernel image is available for execution on the device

Did i miss something? has anybody managed to circumvent such an issue?
If anybody needs further information I can provide a minimal running code example. I also experienced the same issue for another old metric, I described it in more details on stackoverflow.

https://github.com/ufal/MTEQA

Pulling image danieldeutsch/lewis2020:1.0 runs into an error:

docker: failed to register layer: Error processing tar file(exit status 1): lchown /app/bart.large.cnn: invalid argument.

When the pre-trained BART models are untarred during building the Docker image, there was a warning about being unable to change the ownership of the files. I don't remember how that issue was resolved, but I think it's related to this current error.

pytest is currently pinned to exactly 6.2.4, which makes it a little tough to use packages (like qaeval) that depend on sacrerouge and therefore repro in different environments. Could this versioning be relaxed to make it easier to install repro in different environments? e.g. pytest>=6.2.4. Another option would be to move it to the test_requirements of the setup file (or move it under some extra_requirements named dev or test or w/e), assuming it's not needed for actually running any metrics.