microsoft / toga Goto Github PK

License: MIT License

Python 98.71% Shell 1.29%

toga's Introduction

ToGA Artifact

This repository contains the replication artifact for TOGA: A Neural Method for Test Oracle Generation to appear in ICSE 2022.

Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as documentation, bug finding, and preventing regressions. In particular, unit tests document a unit’s intended functionality. A test oracle, typically expressed as a condition, documents the intended behavior of the unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it has to capture the intended functionality and not the implemented functionality. In our paper, we propose TOGA (Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method.

Our artifact reproduces the results for all RQs in the paper's evaluation. The artifact includes source code and download links for datasets and models produced in the paper. We assume basic unix familiarity and ability to run python. The artifact is also available as a docker image for linux.

Docker Setup

For an easy setup, we recommend our docker container that includes all data, pretrained models, and source. Otherwise, follow the setup instructions in the next section.

First, pull the docker image: docker pull edinella/toga-artifact

Connect to it: docker run -i -t edinella/toga-artifact

Then, setup some environment variables: export PATH=$PATH:/home/defects4j/framework/bin export ATLAS_PATH=/home/icse2022_artifact/data/atlas---deep-learning-assert-statements/

Setup

Requirements: python3.9, git lfs

First, clone this repo and install the dependencies:

cd toga/
git lfs pull
pip install -r requirements.txt
git clone https://gitlab.com/cawatson/atlas---deep-learning-assert-statements.git
export ATLAS_PATH=<path_to_atlas...>/atlas---deep-learning-assert-statements/

Tree Sitter setup (optional):

Paring and extracting new sets of unit test inputs requires the tree sitter java grammar. See https://github.com/tree-sitter/py-tree-sitter for instructions on building a tree sitter grammar and use 'vendor/tree-sitter-java'.

Once the grammar is built in a my-languages.so file, place it in /tmp/tree-sitter-repos/my-languages.so

A prebuilt my-languages.so for linux is provided in lib/tree_sitter.

Defects4j setup (optional):

If you want to build and execute defects4j tests, defects4j must be installed.

Requirements:

sudo apt install libdbi-perl
sudo apt install openjdk-8-jdk
sudo apt install libdbd-csv-perl

Models

If you're using our docker image, the pretrained models are in:

icse2022_artifact/model/assertions/pretrained/
icse2022_artifact/model/exceptions/pretrained/

Otherwise, to install our exception and assertion pretrained models, download from: https://drive.google.com/drive/folders/1dZDxu92rZzB_LEwnAkkiy3DblxMJ6nUT?usp=sharing Put them in model/exceptions/pretrained/pytorch_model.bin and model/assertions/pretrained/pytorch_model.bin respectively. The models can also be downloaded from the artifact on zenodo: https://zenodo.org/record/6210589

To train your own model, run

cd model/exceptions/
bash run_train.sh

cd model/assertions/
bash run_train.sh

Datasets - Preprocessed

Our approach is trained and evaluated on three datasets. The first, Atlas* is an adaption of the Atlas dataset. The preprocessed datasets Atlas* and Methods2Test* are included as data/<DATASET>_star.tar.gz files and be accessed by:

cd data
tar xzf atlas_star.tar.gz
tar xzf methods2test_star.tar.gz

The third dataset is our test dataset generated from Evosuite tests. This is located in data/evosuite_tests.tar.gz. Since the dataset size is very large (>400k tests), we provide two smaller sample datasets that can be used to reproduce the bug counts result and false positive rate result respectively from Table 3 for Our Approach in data/evosuite_reaching_tests.tar.gz and data/evosuite_5project_tests.tar.gz. reaching_tests contains bug-reaching tests only, while 5project_tests contains the tests generated for the same 5 defects4j projects used in [1]'s evaluation.

To access the evosuite test datasets run:

cd data
tar xzf evosuite_reaching_tests.tar.gz
tar xzf evosuite_5project_tests.tar.gz
tar xzf evosuite_tests.tar.gz

Evaluation

In our paper, we evaluate three research questions (RQs).

The following commands assume you are in the root of this directory.

RQ1: Is our grammar representative of most developer-written assertions? To evaluate this research question: cd eval/rq1 && python rq1.py
RQ2: Can we infer assertions and exceptional behavior with high accuracy?

Exception Inference:

To reproduce the exception results shown in table 1 for TOGA Model, run:

cd eval/rq2/exception_inference
bash rq2.sh

This script uses the pretrained exception model to predict whether a test is expected to trigger an exception or not, evaluated for accuracy and f1 score on the methods2test_star dataset. Note that the model used in the artifact has been retrained, so the results are slightly different from the submission (accuracy=85% instead of 86%, f1 score is 0.40 instead of 0.39).

To reproduce the weighted coin experiment in table 1, run:

cd eval/rq2/exception_inference
python coin.py

Assertion Inference: To reproduce the exception results shown in table 2 for TOGA Model, run:

cd eval/rq2/assertion_inference
bash rq2.sh

This script uses the pretrained assertion model to predict an assertion given a test prefix and method under test's signature. we evaluate for accuracy and f1 score on the atlas_star dataset.

RQ3: Can we catch bugs with low false alarms?

To reproduce the results shown in Table 3 for Our Approach, run toga.py on either the bug-reaching inputs (for bug results) or 5 project sample (for false positive rate). By default, the toga tool will use the test metadata labels to evaluate oracles predicted by the models and print results. The tool also generates a predicted_oracles.csv file that can be used to generate executable test suites.

Note that we have improved our implementation since the submission and now find 4 additional bugs (58 total instead of 54) with a lower FP rate (22% instead of 25%).

To faciliate faster evaluation, the toga.py will automatically check its predicted oracles against labels included in its metadata input. This can save time since generating and executing test suites is potentially very time consuming. Note that toga will overestimate the FP rate when checking against labels, so the False Positive rate on generated tests will be lower.

To validate the results, use the eval/rq3/rq3.sh to generate and run test suites from the toga generated oracles.

To reproduce table 3 bug result by running only bug-reaching tests (this will not reproduce the FP rate, which requires running on all of the tests):

python toga.py data/evosuite_reaching_tests/inputs.csv data/evosuite_reaching_tests/meta.csv

To reproduce table 3 false positive rate result on a 5 project sample (2+ hour runtime):

python toga.py data/evosuite_5project_tests/inputs.csv data/evosuite_5project_tests/meta.csv

Both bug and FP rate results on the entire dataset (potentially 12+ hour runtime):

python toga.py data/evosuite_5project_tests/inputs.csv data/evosuite_5project_tests/meta.csv

References

Tufano, Michele, et al. "Unit Test Case Generation with Transformers." arXiv preprint arXiv:2009.05617 (2020).

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

toga's People

Contributors

Stargazers

Watchers

Forkers

soneyahossain test-mass-forker-org-1 nashid kleinercubs hzlujunyi jb1ove chan0415 sungzgithub khandakerrahin darthdaver henryhuang1213

toga's Issues

One question about TOGA's implementation in `data/atlas_star_datagen.py`

Hi,
There is a regular expression in "toga/data/atlas_star_datagen.py"(Line 7):

split_re = re.compile("\"<AssertPlaceHolder>\" ;[ }]*")

I think you use this regular expression to separate a test method and a focal method in ATLAS' dataset. If test methods have statements or any other tokens following the marker "<AssertPlaceHolder>", these test methods cannot be matched by this regular expression. I think the correct version of Line 7 in atlas_star_datagen.py should be:

split_re = re.compile("\"<AssertPlaceHolder>\" .*?;( })+")

Could you let me know at your earliest convenience whether the above is my misunderstanding?
Thanks!

How to use TOGA on my own project/tests?

Dear @elizabethdinella,

Let me say you've done a very nice and interesting work and, of course, thank for sharing this artifact with the community.

Quick question, how would one use TOGA on their own project / tests? If possible, a step-by-step on how to execute the many scripts would be great.

--
Best,
Jose

Running TOGA with an assertless dataset

When running TOGA with the provided datasets, for example the evosuite_reaching_tests, TOGA generates two pairs of files:

except_model_inputs.csv and exception_preds.csv;
assert_model_inputs.csv and assertion_preds.csv;

For my project, I am trying to use TOGA on a dataset similar to ATLAS' dataset, where all of the assertions have been replaced by a placeholder, for example, something like this:

public void testIssue705() throws Exception {
    Issue705Bean input = new Issue705Bean(""key"", ""value"");
    String json = MAPPER.writeValueAsString(input);
    // TEST ORACLE
}

where // TEST ORACLE would be our placeholder (this can be replaced by something else too).

Problem is, when i run TOGA with a dataset like this, it fails to write something in the assert_model_inputs.csv (this file will only contain the header), which leads to not being able to generate the assertion_preds.csv and, consequently, fail because it can not find this same file.
This leads to the following stacktrace:

preparing assertion model inputs

// Defects4J: flaky method
// public void testBug2849731() {
// TestIntervalCategoryDataset d = new TestIntervalCategoryDataset();
// d.addItem(2.5, 2.0, 3.0, "R1", "C1");
// d.addItem(4.0, 0.0, 0.0, "R2", "C1");
// ,
// DatasetUtilities.iterateRangeBounds(d));
// }
/**
 * Another test for bug 2849731.
 */
public void testBug2849731_2() {
    XYIntervalSeriesCollection d = new XYIntervalSeriesCollection();
    XYIntervalSeries s = new XYIntervalSeries("S1");
    s.add(1.0, Double.NaN, Double.NaN, Double.NaN, 1.5, Double.NaN);
    d.addSeries(s);
    Range r = DatasetUtilities.iterateDomainBounds(d);
    // TEST ORACLE
}

[]
05/20/2022 21:39:40 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
Some weights of the model checkpoint at microsoft/codebert-base were not used when initializing RobertaForSequenceClassification: ['pooler.dense.weight', 'pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.out_proj.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
05/20/2022 21:39:48 - INFO - __main__ -   Training/evaluation parameters Namespace(data_dir='./', model_type='roberta', model_name_or_path='microsoft/codebert-base', task_name='assertion_classifier', output_dir='./models/assertion_classifier', config_name='', tokenizer_name='', cache_dir='', max_seq_length=200, do_train=False, do_eval=False, do_predict=True, evaluate_during_training=False, do_lower_case=False, per_gpu_train_batch_size=32, per_gpu_eval_batch_size=256, gradient_accumulation_steps=1, learning_rate=1e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=8.0, max_steps=-1, warmup_steps=0, logging_steps=50, save_steps=50, eval_all_checkpoints=True, no_cuda=False, overwrite_output_dir=True, overwrite_cache=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, server_ip='', server_port='', train_file='train.txt', dev_file='valid.txt', test_file='assert_model_inputs.csv', pred_model_dir='model/assertions/pretrained/', test_result_dir='test_results.tsv', n_gpu=0, device=device(type='cpu'), output_mode='classification', start_epoch=0, start_step=0)
testing

05/20/2022 21:39:49 - INFO - __main__ -   Creating features from dataset file at ./

05/20/2022 21:39:49 - INFO - utils -   LOOKING AT ./assert_model_inputs.csv

/home/daniel/work/oracles/toga/model/assertions/utils.py:364: RuntimeWarning: Mean of empty slice.
  print(f'total_samples {len(trunc_b)} total truncations {(trunc_both > 0).sum()}, mean_trunc {trunc_both.mean()}, median {np.median(trunc_both)}, max = {trunc_both.max()}')
/home/daniel/.local/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/daniel/.local/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
Traceback (most recent call last):
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 720, in <module>
    main()
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 715, in main
    evaluate(args, model, tokenizer, checkpoint=None, prefix='', mode='test')
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 236, in evaluate
    eval_dataset, instances = load_and_cache_examples(args, eval_task, tokenizer, ttype='test')
  File "/home/daniel/work/oracles/toga/model/assertions/run_classifier.py", line 421, in load_and_cache_examples
    features = convert_examples_to_features(examples, label_list, args.max_seq_length, tokenizer, output_mode,
  File "/home/daniel/work/oracles/toga/model/assertions/utils.py", line 364, in convert_examples_to_features
    print(f'total_samples {len(trunc_b)} total truncations {(trunc_both > 0).sum()}, mean_trunc {trunc_both.mean()}, median {np.median(trunc_both)}, max = {trunc_both.max()}')
  File "/home/daniel/.local/lib/python3.9/site-packages/numpy/core/_methods.py", line 40, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity



Result of running bash ./model/assertions/run_eval.sh assert_model_inputs.csv
CompletedProcess(args=['bash', './model/assertions/run_eval.sh', 'assert_model_inputs.csv'], returncode=1)



Traceback (most recent call last):
  File "/home/daniel/work/oracles/toga/toga.py", line 218, in <module>
    main()
  File "/home/daniel/work/oracles/toga/toga.py", line 83, in main
    with open("assertion_preds.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'assertion_preds.csv'

I have also tried to first run one of the provided datasets (evosuite_reaching_tests) to have it generate this missing file.
With this, i can run TOGA, however, as expected, the assert_model_inputs.csv is overwritten and left empty.
Also, as may be expected, as the assertion_preds.csv is not generated for the dataset being used, TOGA, despite running, can not infer any assert.

In summary:
I would like to use TOGA on a project that has absolutely no assert statements and have it generate all of the needed asserts, is this in any way possible?

Thank you,
Daniel Bento

javalang.tokenizer.LexerError when parsing input files with escaping characters

Improve the README.md instructions to understand how the input file should be generated.

Test prefixes and focal methods can contain strings with double quote characters. Escaping characters lead to javalang.tokenizer.LexerError. After attempts, I realized a double double quote"" must be used to manage this case.
I would suggest to add information about the input somewhere in the README.md

AttributeError: module 'javalang.parser' has no attribute 'LexerError' while running Toga

Hi,
I am trying to automatize the process of generating test oracles with Toga, for a given project.
However, I got an AttributeError: module 'javalang.parser' has no attribute 'LexerError' when I try to generate oracles for a given class. I have 81 triplets of focal methods, test prefixes, and docstrings (corresponding to 81 rows in the input.csv file). 1 of them raise this exception. I would want to understand the meaning of this error and the cause (it's not clear to me, analyzing the stack trace, the focal method, and the test prefix).

I attach the input.csv and metadata.csv files for replicability. The rows that triggers the error is the following (row 46 of the csv, from line 790 to line 804 - test4545):

"public boolean getUseSeriesOffset() {}","public void test4545() throws Throwable {
    DefaultCategoryItemRenderer defaultCategoryItemRenderer0 = new DefaultCategoryItemRenderer();
    StandardCategoryURLGenerator standardCategoryURLGenerator0 = new StandardCategoryURLGenerator('Null 'stroke' argument.', 'l', 'P');
    defaultCategoryItemRenderer0.setSeriesURLGenerator(1607, (CategoryURLGenerator) standardCategoryURLGenerator0, false);
    assertFalse(defaultCategoryItemRenderer0.getUseSeriesOffset());
}","    /**
     * Returns the flag that controls whether or not the x-position for each
     * data item is offset within the category according to the series.
     *
     * @return A boolean.
     *
     * @see #setUseSeriesOffset(boolean)
     *
     * @since 1.0.7
     */"

The stack trace of the error is the following:

Traceback (most recent call last):
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1345, in parse_block_statement
    statement = self.parse_local_variable_declaration_statement()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1355, in parse_local_variable_declaration_statement
    declarators = self.parse_variable_declarators()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1207, in parse_variable_declarators
    declarator = self.parse_variable_declarator()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1218, in parse_variable_declarator
    array_dimension, initializer = self.parse_variable_declarator_rest()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1230, in parse_variable_declarator_rest
    initializer = self.parse_variable_initializer()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1239, in parse_variable_initializer
    return self.parse_expression()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1773, in parse_expression
    expressionl = self.parse_expressionl()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1788, in parse_expressionl
    expression_2 = self.parse_expression_2()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1814, in parse_expression_2
    expression_3 = self.parse_expression_3()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1870, in parse_expression_3
    primary = self.parse_primary()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1972, in parse_primary
    return self.parse_creator()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 2120, in parse_creator
    arguments, body = self.parse_class_creator_rest()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 2147, in parse_class_creator_rest
    arguments = self.parse_arguments()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 2052, in parse_arguments
    self.accept(')')
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 131, in accept
    self.illegal("Expected '%s'" % (accept,))
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 119, in illegal
    raise JavaSyntaxError(description, at)
javalang.parser.JavaSyntaxError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/darthdaver/Documents/phd/repositories/toga/model/assertion_data.py", line 128, in get_type_info_evo
    test_method_node = parser.parse_member_declaration()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 801, in parse_member_declaration
    member = self.parse_void_method_declarator_rest()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 905, in parse_void_method_declarator_rest
    body = self.parse_block()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1274, in parse_block
    statement = self.parse_block_statement()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1349, in parse_block_statement
    return self.parse_statement()
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 1542, in parse_statement
    self.accept(';')
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 131, in accept
    self.illegal("Expected '%s'" % (accept,))
  File "/Users/darthdaver/Downloads/toga/lib/python3.9/site-packages/javalang/parser.py", line 119, in illegal
    raise JavaSyntaxError(description, at)
javalang.parser.JavaSyntaxError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/darthdaver/Documents/phd/repositories/toga/toga.py", line 232, in <module>
    main()
  File "/Users/darthdaver/Documents/phd/repositories/toga/toga.py", line 86, in main
    method_test_assert_data, idxs = assertion_data.get_model_inputs(tests, methods, vocab)
  File "/Users/darthdaver/Documents/phd/repositories/toga/model/assertion_data.py", line 303, in get_model_inputs
    out = get_type_info_evo(assertion, focal_method, test_method, vocab=vocab)
  File "/Users/darthdaver/Documents/phd/repositories/toga/model/assertion_data.py", line 130, in get_type_info_evo
    javalang.parser.LexerError):
AttributeError: module 'javalang.parser' has no attribute 'LexerError'

If I remove that triplet of focal methods, test prefix and docstring, everything works fine. The problem is that I have different projects to analyze and I want to automatize the process, and process a single triplet at a time is infeasible. I want to understand how can I fix the error, understand if the nature is due to toga or the project.

Thank you in advance!

toga_metadata.csv
toga_input.csv

Error when running toga.py

Hi,

First of all, thanks for publicly sharing this very interesting work.

I am using Python version: 3.9.5. All requirements installed. I can run rq1.py successfully. However, whenever I try to run toga.py, I get the following error. Any help is highly appreciated,

python toga.py data/inputs_omc.csv data/meta.csv

preparing exception model inputs
05/19/2022 10:17:32 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
Some weights of the model checkpoint at microsoft/codebert-base were not used when initializing RobertaForSequenceClassification: ['pooler.dense.weight', 'pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.dense.bias', 'classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
05/19/2022 10:17:34 - INFO - __main__ -   Training/evaluation parameters Namespace(data_dir='./', model_type='roberta', model_name_or_path='microsoft/codebert-base', task_name='exception_classifier', output_dir='./model/exception_classifier', config_name='', tokenizer_name='', cache_dir='', max_seq_length=200, do_train=False, do_eval=False, do_predict=True, evaluate_during_training=False, do_lower_case=False, per_gpu_train_batch_size=32, per_gpu_eval_batch_size=256, gradient_accumulation_steps=1, learning_rate=1e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=8.0, max_steps=-1, warmup_steps=0, logging_steps=50, save_steps=50, eval_all_checkpoints=True, no_cuda=False, overwrite_output_dir=True, overwrite_cache=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, server_ip='', server_port='', train_file='train.txt', dev_file='valid.txt', test_file='except_model_inputs.csv', pred_model_dir='model/exceptions/pretrained', test_result_dir='test_results.csv', print_results=False, n_gpu=0, device=device(type='cpu'), output_mode='classification', start_epoch=0, start_step=0)
testing
Traceback (most recent call last):
  File "/home/user/toga/./model/exceptions/run_classifier.py", line 656, in <module>
    main()
  File "/home/user/toga/./model/exceptions/run_classifier.py", line 649, in main
    model = model_class.from_pretrained(args.pred_model_dir)
  File "/home/user/.local/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1298, in from_pretrained
    raise EnvironmentError(
**OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index', 'flax_model.msgpack'] found in directory model/exceptions/pretrained or `from_tf` and `from_flax` set to False.**
Traceback (most recent call last):
  File "/home/user/toga/toga.py", line 210, in <module>
    main()
  File "/home/user/toga/toga.py", line 46, in main
    results = pd.read_csv('exception_preds.csv', index_col=False)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 482, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
    self._engine = self._make_engine(self.engine)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in __init__
    self._open_handles(src, kwds)
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/parsers/base_parser.py", line 222, in _open_handles
    self.handles = get_handle(
  File "/home/user/.local/lib/python3.9/site-packages/pandas/io/common.py", line 702, in get_handle
    handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'exception_preds.csv'