jaketae / koclip Goto Github PK

View Code? Open in Web Editor NEW

128.0 128.0 16.0 28.57 MB

KoCLIP: Korean port of OpenAI CLIP, in Flax

Home Page: https://tinyurl.com/koclip-app

License: Apache License 2.0

Makefile 0.11% Shell 1.62% Python 98.27%

flax jax openai-clip roberta vision-transformer

koclip's Introduction

Hi there 👋 I'm Jake.

I'm passionate about generative modeling, text-to-speech, NLP, and recommendation systems.

Currently, I'm a senior at Yale studying CS and Math. Previously, I was a

Machine Learning & Software Engineer Intern at Facebook
Machine Learning Engineer Intern at Hugging Face
Machine Learning Research Intern at Neosapience

koclip's People

Contributors

Stargazers

Watchers

Forkers

minsik-ai ampehta tree-park jaekookang jinsu-l mkos11 peternara flying-4-potatoes adventure2165 hizieun facerain tonychae choiking10 caisarl76 dankernel

koclip's Issues

inference.ipynb 마지막 cell

안녕하세요. 좋은 모델 만들어주셔서 감사합니다.

해당 repository의 inference.ipynb의 마지막 cell에 "text" 라는 이름의 객체가 필요해보입니다.

마지막 cell의 맨 첫줄에 아래와 같이 추가하면 될거 같습니다.

기존

inputs = processor(
    text=["소파 위에 고양이", "강아지와 강아지 주인", "쳇바퀴를 달리는 햄스터", "자동차"],
    images=image, 
    return_tensors="jax", # could also be "pt" 
    padding=True
)

...(생략)...

수정 (제안)

text = ["소파 위에 고양이", "강아지와 강아지 주인", "쳇바퀴를 달리는 햄스터", "자동차"]
inputs = processor(
    text=text,
    images=image, 
    return_tensors="jax", # could also be "pt" 
    padding=True
)

...(생략)...

GPU를 이용한 학습 방법 문의

안녕하세요
KOCLIP을 이용하여 저희가 가지고 있는 데이터를 이용하여 학습을 시켜보려고 하는데
제공해주신 run.py, train.sh 를 이용해서 학습을 하면
CPU만 사용을 합니다

os.environ["CUDA_VISIBLE_DEVICES"]
혹은
export CUDA_VISIBLE_DEVICES 를 이용하여 지정을 해 준 후
학습을 하여도 GPU를 사용하지 않고 CPU만 사용을 하여 학습이 진행되고 있습니다.

GPU로 학습을 하는 방법이 따로 있는건지, 아니면 제가 소스를 수정해서 적용 되도록 변경 해야 하는건지
안내 부탁 드리겠습니다.

감사합니다.

학습 관련 문의 2가지.

안녕하세요. KOCLIP 학습 진행 도중 의문점이 생겨 질문을 드립니다.

학습을 진행하면 Loss 와 Eval Loss가 항상 동일합니다. (Learning Rate는 계속 줄어듬)
저의 데이터만 그런게 아니라, 예시로 있는 coco 데이터도 동일합니다.
이게 정상적인 학습이 맞는건지,, 확인 요청 드립니다.

1-1 . KoCLIP 에서 제공해주는 coco 데이터와 train.sh 를 이용하여 학습

Eval Loss 는 Epoch 2부터 계속 동일. 그냥 Loss 는 Epoch 3부터 동일

09/04/2023 11:01:46 - INFO - main - ***** Running training *****
09/04/2023 11:01:46 - INFO - main - Num examples = 413915
09/04/2023 11:01:46 - INFO - main - Num Epochs = 40
09/04/2023 11:01:46 - INFO - main - Instantaneous batch size per device = 64
09/04/2023 11:01:46 - INFO - main - Total train batch size (w. parallel & distributed) = 64
09/04/2023 11:01:46 - INFO - main - Total optimization steps = 258680
Epoch... (1/40 | Loss: 4.158902168273926, Learning Rate: 4.8750189307611436e-05)
Epoch... (1/40 | Eval Loss: 4.158883094787598)
Epoch... (2/40 | Loss: 4.158882141113281, Learning Rate: 4.7500190703431144e-05)
Epoch... (2/40 | Eval Loss: 4.1588826179504395)
Epoch... (3/40 | Loss: 4.158883094787598, Learning Rate: 4.625019209925085e-05)
Epoch... (3/40 | Eval Loss: 4.1588826179504395)
Epoch... (4/40 | Loss: 4.158883094787598, Learning Rate: 4.5000189857091755e-05)
Epoch... (4/40 | Eval Loss: 4.1588826179504395)
Epoch... (5/40 | Loss: 4.158883094787598, Learning Rate: 4.375019125291146e-05)
Epoch... (5/40 | Eval Loss: 4.1588826179504395)

1-2. 준비한 학습용 데이터와 train.sh 를 이용하여 학습

Loss 와 Eval loss 모두 Epoch 1부터 계속 동일 (Epoch 4의 Eval loss 다름)

08/31/2023 15:16:15 - INFO - main - ***** Running training *****
08/31/2023 15:16:15 - INFO - main - Num examples = 2474242
08/31/2023 15:16:15 - INFO - main - Num Epochs = 40
08/31/2023 15:16:15 - INFO - main - Instantaneous batch size per device = 64
08/31/2023 15:16:15 - INFO - main - Total train batch size (w. parallel & distributed) = 64
08/31/2023 15:16:15 - INFO - main - Total optimization steps = 1546400
Epoch... (1/40 | Loss: 4.158883094787598, Learning Rate: 4.8750029236543924e-05)
Epoch... (1/40 | Eval Loss: 4.1588826179504395)
Epoch... (2/40 | Loss: 4.158883094787598, Learning Rate: 4.750003063236363e-05)
Epoch... (2/40 | Eval Loss: 4.1588826179504395)
Epoch... (3/40 | Loss: 4.158883094787598, Learning Rate: 4.625003202818334e-05)
Epoch... (3/40 | Eval Loss: 4.1588826179504395)
Epoch... (4/40 | Loss: 4.158883094787598, Learning Rate: 4.500002978602424e-05)
Epoch... (4/40 | Eval Loss: 4.158883094787598)
Epoch... (5/40 | Loss: 4.158883094787598, Learning Rate: 4.375003118184395e-05)
Epoch... (5/40 | Eval Loss: 4.1588826179504395)
Epoch... (6/40 | Loss: 4.158883094787598, Learning Rate: 4.250002893968485e-05)
Epoch... (6/40 | Eval Loss: 4.1588826179504395)
Epoch... (7/40 | Loss: 4.158883094787598, Learning Rate: 4.125003033550456e-05)
Epoch... (7/40 | Eval Loss: 4.1588826179504395)
Epoch... (8/40 | Loss: 4.158883094787598, Learning Rate: 4.000003173132427e-05)
Epoch... (8/40 | Eval Loss: 4.1588826179504395)
Epoch... (9/40 | Loss: 4.158883094787598, Learning Rate: 3.875002948916517e-05)
Epoch... (9/40 | Eval Loss: 4.1588826179504395)
Epoch... (10/40 | Loss: 4.158883094787598, Learning Rate: 3.750003088498488e-05)
Epoch... (10/40 | Eval Loss: 4.1588826179504395)

이렇게 25 에폭까지 돌리다가 도저히 아닌 것 같아서 종료 했습니다.

configuration 파일 및 weight 파일 저장
현재 train.sh 및 run.py 구성으로 학습을 진행하면
에폭을 돌 때 마다
Configuration saved in /home/test/koclip/checkpoint/config.json
Model weights saved in /home/test/koclip/checkpoint/flax_model.msgpack

이렇게 항상 같은 경로에 파일을 덮어쓰게 되는데
항상 모든 경우에 덮어 쓰게 되는건지 아니면, 최적의 케이스가 발견되면 그때만 덮어쓰게 되는건지 궁금합니다.

답변 주시면 감사하겠습니다!

TypeError: init() got an unexpected keyword argument '_do_init'

Colab 환경에서 실행 시 위와 같은 오류가 납니다.
혹시 어떠한 이유에서 생겨나는 오류이신지 확인해 주실 수 있으신가요?

TypeError                                 Traceback (most recent call last)
[<ipython-input-8-20ea54d41cb6>](https://localhost:8080/#) in <module>
      5 from koclip import load_koclip
      6 
----> 7 model, processor = load_koclip("koclip-base")
      8 

2 frames
[/content/koclip/koclip/model.py](https://localhost:8080/#) in __init__(self, config, input_shape, seed, dtype, **kwargs)
    159             )
    160 
--> 161         module = self.module_class(config=config, dtype=dtype, **kwargs)
    162         super().__init__(
    163             config, module, input_shape=input_shape, seed=seed, dtype=dtype

TypeError: __init__() got an unexpected keyword argument '_do_init'

*** ValueError: You have to specify pixel_values for text embedding

Hi I'm trying to extract text embedding and get the *** ValueError: You have to specify pixel_values error.
Here are the code to reproduce:

    repo = "koclip/koclip-base-pt" 
    model = AutoModel.from_pretrained(repo)
    tokenizer = AutoTokenizer.from_pretrained(repo)
    text_only_inputs = tokenizer(text="test", return_tensors="pt", padding=True)
    model(**text_only_inputs)

Appreciate for any solutions. Thanks!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.