안녕하세요 좋은 모듈 만들어주셔서 감사합니다.
제가 cuda 환경에서 모델을 불러와서

pprint(goemotions(text))

를 실행하면 아래와처럼 에러가 뜨네요.
multilabel_pipeline 에서 device를 -1에서 0으로도 바꿔보고,

import torch
device = torch.device('cuda:0')
model.to(device)

위 처럼 추가 설정도 해보았는데 동일한 에러가 떠서 여기에 여쭤보게 됐습니다.
어떻게 고칠 수 있을까요?

감사합니다.

RuntimeError Traceback (most recent call last)
in
----> 1 pprint(goemotions(df['sentences'][0]))

~/ProjComment/IMDB/GoEmotions-pytorch/multilabel_pipeline.py in call(self, *args, **kwargs)
37
38 def call(self, *args, **kwargs):
---> 39 outputs = super().call(*args, **kwargs)
40 scores = 1 / (1 + np.exp(-outputs)) # Sigmoid
41 results = []

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/transformers/pipelines.py in call(self, *args, **kwargs)
472 def call(self, *args, **kwargs):
473 inputs = self._parse_and_tokenize(*args, **kwargs)
--> 474 return self._forward(inputs)
475
476 def _forward(self, inputs, return_tensors=False):

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/transformers/pipelines.py in _forward(self, inputs, return_tensors)
491 with torch.no_grad():
492 inputs = self.ensure_tensor_on_device(**inputs)
--> 493 predictions = self.model(**inputs)[0].cpu()
494
495 if return_tensors:

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
--> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)

~/ProjComment/IMDB/GoEmotions-pytorch/model.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
25 labels=None,
26 ):
---> 27 outputs = self.bert(
28 input_ids,
29 attention_mask=attention_mask,

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
--> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
724 head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
725
--> 726 embedding_output = self.embeddings(
727 input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
728 )

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
--> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
172
173 if inputs_embeds is None:
--> 174 inputs_embeds = self.word_embeddings(input_ids)
175 position_embeddings = self.position_embeddings(position_ids)
176 token_type_embeddings = self.token_type_embeddings(token_type_ids)

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
530 result = self._slow_forward(*input, **kwargs)
531 else:
--> 532 result = self.forward(*input, **kwargs)
533 for hook in self._forward_hooks.values():
534 hook_result = hook(self, input, result)

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
110
111 def forward(self, input):
--> 112 return F.embedding(
113 input, self.weight, self.padding_idx, self.max_norm,
114 self.norm_type, self.scale_grad_by_freq, self.sparse)

/home/ubuntu/anaconda3/envs/GoEmotions-pytorch/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
1482 # remove once script supports set_grad_enabled
1483 no_grad_embedding_renorm(weight, input, max_norm, norm_type)
-> 1484 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
1485
1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

Hi,
Thank you for your great work on GoEmotions-pytorch!
I am trying to use your code to generate models using either bert-mini, bert-small or bert-tiny for faster predictions.
I changed the file original.json by setting model_name_or_path to prajjwal1/bert-mini for example and I run python3 run_goemotions.py --taxonomy original
It works and the new model is a bit faster than the one using bert-base. However, I was wondering if I need to also change the tokenizer_name_or_path to a different value. The original value is "monologg/bert-base-cased-goemotions-original". Any thoughts on how to get a tokenizer based on bert-mini?

Many thanks!
Chedia

Hi!
Would you be able to help me and provide me with the www.syndicai.co file to deploy the project there?
That would really help me :) Thanksa. lot!

Hi,

Thanks so much for this repo and please forgive me if this is trivial, I've been trying for a little while now to run the model on Google Colab. I'm running into two separate issues, which I think may be linked. The first, is that if I load the model in a GPU runtime the model defaults to the 'cpu'.

After running:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model.to(device)

There's an error when trying to run goemotions(texts):
"RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select".

Second, when trying to run goemotions over more than a few thousand rows on Colab in a high-RAM runtime environment, I run into an out of memory error. I'm wondering if this is a problem with batching in the data loader? I'll be looking for solutions & hope to close this issue myself, but in the meantime any help is much appreciated, thanks!

I am looking into loading the model and the tokenizer after it has been trained on a custom dataset. After training, I am able to produce pytorch_model.bin, config.json, tokenizer_config.json, special_tokens_map.json, training_args.bin, and vocab.txt for every checkpoint saved.

Is there any script where I can know how to load the saved checkpoints along with the tokenizer just like the example that you have provided here for your pre-trained model

tokenizer = BertTokenizer.from_pretrained("monologg/bert-base-cased-goemotions-ekman")
model = BertForMultiLabelClassification.from_pretrained("monologg/bert-base-cased-goemotions-ekman")

goemotions = MultiLabelPipeline(
    model=model,
    tokenizer=tokenizer,
    threshold=0.3
)

Thanks for the awesome repo!

Thanks for this great work.
I am using transformer v4
I know this is not transformer v2 as requested in the readme file but I cannot install v2.11.0 anymore because there are dependency errors in that version. And I tried v2.4.1 it raise other errors.

In transformer v4, it raises:
Traceback (most recent call last):
goemotions = MultiLabelPipeline(
TypeError: Can't instantiate abstract class MultiLabelPipeline with abstract methods _forward, _sanitize_parameters, postprocess, preprocess

Do you think you can update your code so it can work with the latest Hugginface transformer (v4)?

Please provide the aws link for downloading your finetuned model. Thanks.

Shouldn't sigmoid(multilabel classification) or softmax(multiclass classification) be applied here to get the predictions? Otherwise, later the thresholds are being applied on the logits directly.

my input text is too long, so there's an error: RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:418. how to solve it?

your work is very interesting and valuable.
I would like to know how to get the final hidden representations of sentence to get the vector.
Thanks

Let's say we want to fine-tune the model (any of the taxonomies - ekman/original) on another dataset having a different number of classes for e.g. only positive and negative. Then what's the correct procedure to do it?

Currently, If I prepare the data in the same format (.tsv files with data and labels), and put the labels.txt as having only two classes, the training seems to run. But is this way correct? Or any other changes need to be made inside the model training?