eval_bleu with pretrained gpt model about avatar HOT 5 CLOSED

Too many information and thus unable to understand the problem. I don't see any eval_blue script too. Please look at the code for other models.

from avatar.

eval_bleu is here: https://github.com/wasiahmad/AVATAR/blob/main/codegpt/run.py

for step, (batch, token_labels) in enumerate(test_dataloader):
        inputs = batch.to(args.device)
        with torch.no_grad():
            beam_size = args.beam_size
            m = torch.nn.LogSoftmax(dim=-1)
            outputs = model(inputs)[1]
            p = []
            zero = torch.cuda.LongTensor(1).fill_(0)
            for i in range(inputs.shape[0]):
                past_hidden = [x[:, i:i + 1].expand(-1, beam_size, -1, -1, -1) for x in outputs]

I would like to evaluate the model, but the error TypeError: tuple indices must be integers or slices, not tuple occurs on the line past_hidden = [x[:, i:i + 1].expand(-1, beam_size, -1, -1, -1) for x in outputs]

from avatar.

I am not sure. The CodeGPT codebase is supposed to work fine. I am not sure why you are getting this error. I will run again in my environment to see if CodeGPT training and evaluation work correctly.

from avatar.

The problem is that GPT-2 model's output format has changed in the newer versions of Huggingface's transformers library – thus, it needs to apply torch.stack() to the output.

from avatar.

I see, you can modify the code to be compatible with the newer versions of the transformers API. I am closing this issue for now.

from avatar.