Giter Club home page Giter Club logo

Comments (17)

maxjeblick avatar maxjeblick commented on August 14, 2024 6

Thanks for the fix!
I tested udop-dual-large-224 with the code below (where I have downloaded model weights before), and it now works.

from core.models import UdopDualForConditionalGeneration, UdopUnimodelForConditionalGeneration, UdopConfig, \
    UdopTokenizer


config = UdopConfig.from_pretrained("../udop-dual-large-224")
tokenizer = UdopTokenizer.from_pretrained("../udop-dual-large-224")
model = UdopDualForConditionalGeneration.from_pretrained("../udop-dual-large-224")

from i-code.

zinengtang avatar zinengtang commented on August 14, 2024 4

Yeah the configurations should be identical to t5 (tokenizer) or can be initialized by default parameters.
We will add our own configuration files when we add the model to Huggingface Transformer.

from i-code.

ziyi-yang avatar ziyi-yang commented on August 14, 2024 3

Thanks again for your great interest in our work, also we appreciate huggingface team's efforts. We'll fix the issue on our side ASAP.

from i-code.

logan-markewich avatar logan-markewich commented on August 14, 2024 2

@go2carter
You might have to add a few keys to the config manually. Check out the values from this config

This is from the PR to add the model to huggingface. Unless you are in a rush, it might be best to wait for the model to be added to huggingface. Everything will work much smoother then :)

from i-code.

go2carter avatar go2carter commented on August 14, 2024 2

@logan-markewich
Thank you! I'll try that out when I get the opportunity. Not in a rush, but I'm excited to try out this work :)

from i-code.

zinengtang avatar zinengtang commented on August 14, 2024 2

I have made a PR on fixing the issues and updated the model checkpoint to huggingface style checkpoints/configs. Let me know if the issue is still not addressed.

from i-code.

logan-markewich avatar logan-markewich commented on August 14, 2024 1

@raghavanone according to the README, the vision decoder weights will only be released as part of an azure API. Therefore, not exactly openly available.

from i-code.

ziyi-yang avatar ziyi-yang commented on August 14, 2024 1

Yes, running inference with the provided model weights is doable.

from i-code.

raghavanone avatar raghavanone commented on August 14, 2024

@zinengtang I am trying to add the model to huggingface here . But the model weights are incomplete with out vision decoder weights.

from i-code.

maxjeblick avatar maxjeblick commented on August 14, 2024

It's still unclear to me how to use the model.
I tried initializing as

import torch

from core.models import UdopDualForConditionalGeneration, UdopUnimodelForConditionalGeneration, UdopConfig, \
    UdopTokenizer

MODEL_CLASSES = {
    'UdopDual': (UdopConfig, UdopDualForConditionalGeneration, UdopTokenizer),
    'UdopUnimodel': (UdopConfig, UdopUnimodelForConditionalGeneration, UdopTokenizer),
}

config = UdopConfig.from_pretrained("t5-large")
# also tried config = AutoConfig.from_pretrained("t5-large")

tokenizer = UdopTokenizer.from_pretrained("t5-large")
model = UdopUnimodelForConditionalGeneration.from_pretrained("t5-large")
# also tried model = UdopUnimodelForConditionalGeneration(config)
model.load_state_dict(
    torch.load("models/UdopUnimodel-Large-224/pytorch_model.bin"))

which gives

AttributeError: 'T5Config' object has no attribute 'truncate_encoder_after_layer'

from i-code.

go2carter avatar go2carter commented on August 14, 2024

I fixed that by fixed the lines in "udop_dual.py" and "udop_unimodel.py" to:

        if self.is_decoder:
            self.num_layers = (
                config.truncate_decoder_after_layer
                    if (hasattr(config, 'truncate_decoder_after_layer') and config.truncate_decoder_after_layer)
                    else config.num_layers
            )
        else:
            self.num_layers = (
                config.truncate_encoder_after_layer
                    if (hasattr(config, 'truncate_encoder_after_layer') and config.truncate_encoder_after_layer)
                    else config.num_layers
            )

from i-code.

go2carter avatar go2carter commented on August 14, 2024

but then I run into the following error for both models:
AttributeError: 'UdopConfig' object has no attribute 'data_dir'

from i-code.

ziyi-yang avatar ziyi-yang commented on August 14, 2024

Right the current checkpoint doesn't contain the vision decoder part. We are still in the discussion of the most appropriate way of open sourcing that part.

from i-code.

ziyi-yang avatar ziyi-yang commented on August 14, 2024

It's still unclear to me how to use the model. I tried initializing as

import torch

from core.models import UdopDualForConditionalGeneration, UdopUnimodelForConditionalGeneration, UdopConfig, \
    UdopTokenizer

MODEL_CLASSES = {
    'UdopDual': (UdopConfig, UdopDualForConditionalGeneration, UdopTokenizer),
    'UdopUnimodel': (UdopConfig, UdopUnimodelForConditionalGeneration, UdopTokenizer),
}

config = UdopConfig.from_pretrained("t5-large")
# also tried config = AutoConfig.from_pretrained("t5-large")

tokenizer = UdopTokenizer.from_pretrained("t5-large")
model = UdopUnimodelForConditionalGeneration.from_pretrained("t5-large")
# also tried model = UdopUnimodelForConditionalGeneration(config)
model.load_state_dict(
    torch.load("models/UdopUnimodel-Large-224/pytorch_model.bin"))

which gives

AttributeError: 'T5Config' object has no attribute 'truncate_encoder_after_layer'

@zinengtang Can you take a look at this?

from i-code.

go2carter avatar go2carter commented on August 14, 2024

Right the current checkpoint doesn't contain the vision decoder part. We are still in the discussion of the most appropriate way of open sourcing that part.

I see. We should be able to run inference then still with the provided model weights--correct?

from i-code.

go2carter avatar go2carter commented on August 14, 2024

I've also been having trouble loading the model to test it out. When I tried, @maxjeblick 's code above with my fixes, then I get: AttributeError: 'T5Config' object has no attribute 'max_2d_position_embeddings'

from i-code.

go2carter avatar go2carter commented on August 14, 2024

awesome! Excited to try it out when I get a chance

from i-code.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.