backprop-ai / backprop Goto Github PK

Which clip model and weights are used, is it the same as in "openai/clip-vit-base-patch32"?

Hello Backprop Team!

Great job on the library.

I was trying to replicate the Generate Questions Fine-tuning example: https://github.com/backprop-ai/backprop/blob/main/examples/Finetuning_GettingStarted.ipynb

However, I'm facing the following error:

Exception: You need a cuda capable (Nvidia) GPU for fine-tuning.

When I add the following line to my code while calling the TextGeneration Model
device="cuda"
the error changes to No CUDA GPUs are available

I'm using the following AWS EC2 instance which claims to have NVIDIA CUDA: https://aws.amazon.com/marketplace/pp/Amazon-Web-Services-AWS-Deep-Learning-Base-AMI-Ubu/B07Y3VDBNS#pdp-overview

Moreover, when I run the command nvcc --version, I see the following output:

Please help. Where am I going wrong?

Best,
Karan

Hi,

I am running the emotion service on cuda and I get this following RuntimeError:

I am sending a batch_size of 256 to extract emotions.
and generate(text, do_sample=False, max_length=512)
Is this something related to max seq length?

File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/conda/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/conda/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/root/sales_emotion/main.py", line 34, in task_level_emotion
response = emotion_task_to_db(task_id)
File "/root/sales_emotion/src/services/emotion_task_level.py", line 67, in emotion_task_to_db
df_insert = task_level_emo(task_id)
File "/root/sales_emotion/src/services/emotion_task_level.py", line 40, in task_level_emo
emotion_labels.extend(kiri.emotion(text))
File "/root/sales_emotion/core.py", line 50, in emotion
return self._emotion(input_text)
File "/root/sales_emotion/models/tasks/emotion.py", line 42, in call
return self.model.emotion(text)
File "/root/sales_emotion/models/custom_models.py", line 25, in emotion
return self.generate(text, do_sample=False, max_length=512)
File "/root/sales_emotion/models/models.py", line 138, in generate
features = self.tokenizer(text, return_tensors="pt")
File "/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2356, in call
**kwargs,
File "/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2426, in encode_plus
**kwargs,
File "/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 465, in _encode_plus
**kwargs,
File "/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 372, in _batch_encode_plus
pad_to_multiple_of=pad_to_multiple_of,
File "/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 325, in set_truncation_and_paddi
ng
self._tokenizer.no_truncation()
RuntimeError: Already borrowed

Please help.

Chirag Sanghvi

Hi, thank you for the amazing work.

Could you please add the example notebook/code for fine-tuning of the CLIP model?

How can I finetune the CLIP model for the image classification task? could you please add an example notebook other than EfficientNet finetuning?

Hi

I am trying to run this code below on GPU, where should I specify device and what is the command like ?

device='gpu' or device='cuda' and where should I be mentioning it ?

This is your old code bit :

from transformers import T5ForConditionalGeneration, T5Tokenizer

MODEL = "kiri-ai/t5-base-qa-summary-emotion"
TOKENIZER = "t5-base"

def generate(input_text, model_name: str = None, tokenizer_name: str = None):
    # Refer to global variables
    global model
    global tokenizer
    # Setup
    # Initialise model
    if model == None:
        # Use the default model
        if model_name == None:
            model = T5ForConditionalGeneration.from_pretrained(MODEL)
        # Use the user defined model
        else:
            model = T5ForConditionalGeneration.from_pretrained(model_name)

    # Initialise tokenizer
    if tokenizer == None:
        # Use the default tokenizer
        if tokenizer_name == None:
            tokenizer = T5Tokenizer.from_pretrained(TOKENIZER)
        # Use the user defined tokenizer
        else:
            tokenizer = T5Tokenizer.from_pretrained(tokenizer_name)

    is_list = False
    if isinstance(input_text, list):
        is_list = True

    features = tokenizer(input_text, padding=True, return_tensors='pt')
    tokens = model.generate(input_ids=features['input_ids'],
                            attention_mask=features['attention_mask'], max_length=512)
    if is_list:
        return [tokenizer.decode(tokens, skip_special_tokens=True) for tokens in tokens]
    else:
        return tokenizer.decode(tokens[0], skip_special_tokens=True)

def process_item(item):
    return f"emotion: {item}"

def emotion(input_text, model_name: str = None, tokenizer_name: str = None):

    if isinstance(input_text, list):
        input_text = [process_item(item) for item in input_text]
    else:
        input_text = process_item(input_text)

    return generate(input_text, model_name=model_name,
                    tokenizer_name=tokenizer_name)

Best,
Chirag

Hello, whenever i go to clip.backprop.co and put a picture with some labels and click on "Predict image" i get "something went wrong, try again". Why is that?

Thank you

Hello, I tried finetuning Image-Text Vectorizer CLIP model using above approach. But I get stuck with the error -

Link to full code - Colab

What I need is something which gives cosine similarity between an image and a text, shall I finetune with triplet, or with cosine similarity? if its cosine similarity, then how will I get those cosine similarity?

The triplet variant takes text and image and gives one normalised vector, I am bit confused because I thought it would give a cosine similarity.

Hi,

First of all thanks a ton for this repository, the results that I am looking for is quite promising without any down-stream task training. :)

Are you planning to publish the paper/documentation on the data T5 is trained for emotion ?

Best,
Chirag Sanghvi

Hello, I have multiple PDFs and I convert them from pdftotext library, I have 2 questions:

1. QA: If I pass the entire PDF object it shows unknown whereas checking each page shows the correct result i.e for page in pdf works if page is context for QA whereas pdf itself as the context shows 'unknown' as result
2. Is there a way to use multiple docs/PDF in some way and generate the result based on the most likely answer?