jun-jie-huang / exeds Goto Github PK

Source code and data for our paper at DaSH@EMNLP-22: "Execution-based Evaluation for Data Science Code Generation Models"

License: MIT License

Python 45.53% Shell 0.74% Makefile 0.36% Jinja 3.33% CSS 21.44% Batchfile 0.31% Jupyter Notebook 17.12% HTML 11.15% Smarty 0.01%

exeds's People

Contributors

Stargazers

Watchers

Forkers

kpouianou

exeds's Issues

no codex or other LLM reproduceable code

I wonder what steps I should follow to reproduce your Codex results.
Also, as the Codex has been closed, I would appreciate it if you provide a script to apply some large language models (e.g., gpt-3.5-turbo a.k.a. ChatGPT) to your test set.

The checkpoint weights

do you know where I can find the checkpoint weights for the trained PyMT5 and JupyT5? Thanks.

Help with the code snippet generation part

Could you help me with this step? which command(s) can I run to do this part of the README: you can generate the code snippets for each example of ExeDS with your model.?

Error while initializing BPE

Hi again,

I ran this command in Section 2.2 and I get the error in the picture below. Could you please advise?

python preprocess.py \
            --split dev \
            --file_name exeds_dev.json \
            --do_fairseq_tokenization \
            --do_gptneo \
            --token_type token \
            --context_range 3 \
            --max_code_cell_tokens 200 \
            --max_md_cell_tokens 200 \
            --max_ctx_cell_tokens 900