uark-aicv / vlcap Goto Github PK

View Code? Open in Web Editor NEW

28.0 3.0 5.0 94.27 MB

[ICIP 2022] VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning

Home Page: https://ieeexplore.ieee.org/document/9897766

Python 24.01% Jupyter Notebook 75.60% Shell 0.39%

transformer video-captioning vision-and-language contrastive-learning

vlcap's People

Contributors

Stargazers

Watchers

Forkers

joskid teytaud yqgao716 daweiro huaish

vlcap's Issues

Cannot get the same score as the score in the paper

In the paper https://arxiv.org/pdf/2206.12972v2.pdf, this model gets:
B@4: 14.00
METEOR: 17.78
CIDEr: 32.58
ROGUEr: 36.37

However, I trained this model on my machine without changing any configuration and only got:
B@4: 12.11
METEOR: 16.51
CIDEr: 31.29
ROGUEr: 32.52
And I trained this model on V100 GPU

Should I change any configuration to get the highest score?

Thank you for providing this great code!

bash scripts/train.sh anet true generates

Hi,
after running bash scripts/train.sh anet true generates
there is a long list of

---------------------------------------------------------
>>>>>>>> Running training on anet dataset
>>>>>>>> using lang feature: true
---------------------------------------------------------
2022-11-28 11:21:20,576: Mode train
2022-11-28 11:21:20,576: Loading data from /home/iqa000/Workspace/VLCAP-main/densevid_eval/anet_data/train.json
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10009/10009 [00:00<00:00, 1445768.80it/s]
2022-11-28 11:21:20,615: Loading complete! 10009 examples
  0%|                                                                                                                                                                                                        | 0/10009 [00:00<?, ?it/s]missing
missing
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10009/10009 [00:00<00:00, 148278.61it/s]
Missing 20020 features (clips/sentences) from 10009 videos
Missing {'tSk1GWyofaU', 'zu960Glpzo4', 'Vx6vP1oxiAg', 'FMlWHXByLL0', 'evj6y2xZCnM', 'zBmVL3I3nFU', '2Is_nJdG2to', 'cSfs5ht9sro', 'HKkzII7ap7E', 'ULBhK8jXNws', 'c1eUdyyT4zg', '7AkyOhKkT6g', 'XP5Oqr1giQ4', 'VcthLhKIntA', 'uOmCwWVJnLQ', 'HeMpg3SAUUs', 'hYBctolxeqQ', 'XSNenkxgryQ', '2x-Xqt98Ek4', 'ot7hBY4lQ2c', 'SSLcbqaBiRM', 'VdeYnCIbRJ4', 'j1oB2NAlYsQ', 'w2fsq9BOoZo', 'n6k21NjvqXE', '9mF5s6_dTlk', 'weB3srg6o4c', 'WmabLngcvas', 'Kp7pUEKrb8Q', 'OdLcbH2H_zI', 'BERvPz1e_AU', 'a25vC5zsf6A', '1dM62Xpm9Ns', 'KfP205pf7PU', 'pu-2w-UxdYg', 'KKbfCtmIE0o', 'Lan3mtnCmlw', '8SCg3toperM', 'J6ScF5n_Cug', 'sCzauf2u4dc', '0AjYz-s4Rek', 'L1XpfS1RCzE', 'qkN9uA8izVE', 'r40TuTkt9y4', '2j-DRUk2yCs', 'u7dfBgc_SqU', '5HCYb6qfkdk', 'nVk5nIE-6bM', 'eGLD-0b1LV0', 'iUe1t0sN4Jo', '3pBldeB3uaE', 'n1KeC6NXPUA', 'Z1POv1Qeno0', '_NwkwvaC7Bg', '5zPTTiJiXUY', 'w5J3Gt5WLwU', '9XyrLUWZl40', 'ru7UAr2488M', '7MWDmMh3zyA', 'HDHS_7pOiDk', '_9h6NBOPTy8', '-jl_v7zi17A', '33SI8z8PovA', 'UB2GzjNzo3M', 'ofrX4WyAM-0', 'JOYduGqZSRc', 'w6Avae5on_0', 'KYtV2vpwuVw', '-NndIs9BaS4', 'qNxA4UTadGo', 'w_wIOJrztdU', 'mgmwdQixDXY', '9ku5v_hSVMw', 'RgzbNJPchqc', 'uUzmPV8Vgqg', 'auxBRPzLiIo', 'VwclmKWo_-M', 'D9eo9NfFhkg', 'G5ueYVLGtm8', 'pwPid8YHHpU', 'F6cNWYlfUs8', '_cZD6JN-SYg', 'VDX1IQnUMgo', '_vbwjI1QA7g', 'P9jIpcRGeOk', '6hsOVkC7hxA', 'yeUuZ9vk5gE', 'VEihQG2UWKE', 'O8vPTn6Ho7w', 'TTDruR5Vin4', 'PbzmcZ_IORE', '7lNAmkaMyyg', 'ABB755sPZfY', 'ui_CNb4FUtQ', 'fLvPz8W00l4', 'Epl3pExUuNs', 'zVMDHCnT-d4', 'zD_wAe6Eoxc', '2PAVJbmj2lQ', '8AP2he781Cw', 'hFtmkU7wdx4', '3zDw5mwGIW0', 'Y-1QkIGm81w', 'ucEqZtmQS-0', '7EPzlmJ25dA', 'qi_6u0mMJQM', 'W_iKlOPSDos', 'M2ntILX6VP0', 'QkqsI11OtC8', 'nwznKOuZM7w', 'b82y7f7TFbw', 'tRatWgaZ-a0', '_--nxrRXdPg', 'OhXBMlKOHMI', 'kt_sGN-1prU', 'nnWJGghixr0', 'tydn-vo3DaY', 'sQo4gMcgfT4', 'O4P07fipvIA', '__mIAEE03bE', 'B5Zi054Fa5k', '9g-5J05BIiQ', 'FkSf3pxra3M', 'keFBEoBy0zY', 'zuqNxHmtBD8', 'bJ1vEQKX-hE', 'm8SFyH4vhik', 'diBZlwUO8rc', '_4CLYKFzmoY', 'KFIxTdJtXAE', 'iDz8nKDpumY', '3X2CY79a0X8', 'gdr6iVHHYcU', 'PlAVnu-ueM4', '06eyqLosXjU', 'qumU7AgV3Mk', 'vu65aIIJHtU', 'Fdjw9ld-hbA', 'dyLGepr7VR0', 'zvAlL20-K4w', '6SHSstpZN1I', '0ZzKrBk1ac8', 'ThYidZUtnuo', 'mpyN1mrMl3U', '0jBwj0bfZ3Y', 'fJ7gcHxxJMM', 'm2DOej6tPNs', 'xIU6DO35R_c', 'YuCMWTdK_DY', 'QPKJDlQSO6c', 'Y1j_e1DXW6I', 'L9MTwigRhmk', 'ufK2mbJI0to', 'PNdG3SUdJzc', 'mG8h5rX3OnU', 'FXl3qRRs9jw', 'vw64k9rIi_g', '4mRdgV8t4KY', 'hQXWnoipdFE', 'aq41GgfAlDo', 'Tm1ebIrDyz0', '6lYTHj9vImo', '57cM1GcKktw', '09Kr5TQ9DHQ', 'xE43h7Kd9Oc', 'CNH37tJNzFE', 'Bhz-WgJH8R0', '8dXbbJWFEJo', 'yJgC3-t_ciw', 'IjKWgD0y4rc', 
't5Br7yOUe4g', 'q-ID2mgEIow', 'SfiAcQAPpQ8', 'M6yAoJJQvGY', 'Gc1Mk5UyECQ', 'HEfOp_pz_j4', 'aEyTdUOp-qs', '5TV-V6Cxero', 'PXBcPu2_KOo', 'oOnKQgQZOZ0', 'OIA7lPraPSM', 'my4UPLGI6w4', 'deuSw3RnNLU', 'Igm1Mx4Ng1k', 'IDr50VT8BK8', 'uu4_cV49pMI', 'BnkUgUQBED0', 'jsu65VwKf74', '96krk6Ka9Vc', 'b7fs8OAJzQk', 'Y76wuHBZgdU', 'uzXbaoWOm5o', 'mX3gbTBdbKY', 'bj4nkWPdqIY', 'INmaUkmVK24', 'GySHt3Z6Lt4', 'UGCn1zgYboQ', 'x7lP6GKepco', 'IKTYMYu8FFs', '9rW35YTKYq8', 'ykcLgz3DlYg', '9n_cwQLpo_c', '_B3Q8bTJWG4', 'DlE6Rtuo__o', 'J_SD_hhGET8', '0CTnYEE7rdo',

But then it displays the following error.

File "/home/iqa000/Workspace/VLCAP-main/src/train.py", line 684, in
main()
File "/home/iqa000/Workspace/VLCAP-main/src/train.py", line 630, in main
train_loader = DataLoader(train_dataset, collate_fn=collate_fn,
File "/home/iqa000/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 344, in init
sampler = RandomSampler(dataset, generator=generator) # type: ignore[arg-type]
File "/home/iqa000/.local/lib/python3.10/site-packages/torch/utils/data/sampler.py", line 107, in init
raise ValueError("num_samples should be a positive integer "
ValueError: num_samples should be a positive integer value, but got num_samples=0

I am following read.me like a holy thing and there is data also in the data folder. (env, lang, sens)
After running evaluation command bash scripts/translate_greedy.sh anet_re_init_2022_11_22_17_11_43 test
I also get similar type of error. Is there some issue with dataloader?

Also, I noticed that after running
bash scripts/translate_greedy.sh anet_re_init_2022_11_22_17_11_43 test
results/anet_re_*/greedy_pred_val.json json file is note created.

YouCook2 Features

Is it possible to provide the YouCook2 features? Or could you provide the script/code to extract features from the YouCook2 dataset or the configuration of C3D/Clip_b16?

Thank you so much for the fantastic code!

ValueError: num_samples should be a positive integer value, but got num_samples=0

This looks like a simple thing like an argument missing but if you have a suggestion I'm interested.
The command line was << bash scripts/train.sh yc2 true >>.

Traceback (most recent call last):
File "src/train.py", line 706, in
main()
File "src/train.py", line 642, in main
num_workers=opt.num_workers, pin_memory=opt.pin_memory)
File "XXX/.conda/envs/videocap/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 270, in init
sampler = RandomSampler(dataset, generator=generator) # type: ignore[arg-type]
File "XXX.conda/envs/videocap/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 103, in init
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0

How to pre-train matrix Wi by CLIP to obtain image embedding?

24/5000
I'm sorry, I don't quite understand this part, can you elaborate? Thank you！

clip image feature

Thank you so much for the fantastic code!
could you please share the center frame of snippets feature extracted by the image encoder of CLIP if you are convenient

How to change the configs,such as num_epochs and batch_size

Unable to download anet_c3d.zip

Unable to download anet_c3d.zip.
Is there any fix for downloading anet_c3d.zip smoothly? It shows network error after half of the download is complete

Unable to download env feature and lang feature

Hi
Previously i was able to download the files for environment and language features but now the site says
Sorry, the file you requested doesn't exist.

Check that you have the correct URL and that the file exists.

data Preparation

thanks for this fantastic work too.
i am new to the research topic, could you please share some common methods to generate the c3d video feature file (100*2048) and the lang_feature (100*100 of words) and sent_feature (n_sent * 512) if you are convenient

clip 1.0 ?

Thanks for this fantastic work.
The requirements file mentions clip=1.0, but https://pypi.org/project/clip/ mentions versions only up to 0.2.0. So the installation crashes.
Do you have a solution ?
<<
Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement clip==1.0 (from -r VLCAP/condaenv.vopj0xmb.requirements.txt (line 29)) (from versions: 0.0.1, 0.1.0, 0.2.0)
ERROR: No matching distribution found for clip==1.0 (from -r VLCAP/condaenv.vopj0xmb.requirements.txt (line 29))

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.