jayleicn / mtvretrieval Goto Github PK
View Code? Open in Web Editor NEW[ACL 2021] mTVR: Multilingual Video Moment Retrieval
mtvretrieval's People
Contributors
Stargazers
Watchers
mtvretrieval's Issues
A stuck problem during the pre-training
The third bug, an error?
By the way, my computer also encountered a stuck problem during the pre-training and I had to restart it.
But when I checked the log file, I found that it successfully produced the model.ckpt and
en_best_tvr_val_predictions_VCMR_SVMR_VR.json
en_best_tvr_val_predictions_VCMR_SVMR_VR_metrics.json
zh_best_tvr_val_predictions_VCMR_SVMR_VR.json
zh_best_tvr_val_predictions_VCMR_SVMR_VR_metrics.json
en_latest_tvr_val_predictions_VCMR_SVMR_VR.json
en_latest_tvr_val_predictions_VCMR_SVMR_VR_metrics.json
zh_latest_tvr_val_predictions_VCMR_SVMR_VR.json
zh_latest_tvr_val_predictions_VCMR_SVMR_VR_metrics.json
en_inference_tvr_val_None_predictions_SVMR_VCMR_VR.json
en_inference_tvr_val_None_predictions_SVMR_VCMR_VR_metrics.json
en_inference_tvr_val_None_predictions_SVMR_VCMR_VR_nms_thd_0.5.json
I think there may be two reasons for that:
- My gpu and memory resources are insufficient
- There is a bug when producing the following files:
zh_inference_tvr_val_None_predictions_SVMR_VCMR_VR.json
zh_inference_tvr_val_None_predictions_SVMR_VCMR_VR_metrics.json
zh_inference_tvr_val_None_predictions_SVMR_VCMR_VR_nms_thd_0.5.json
Bugs about missing tvr_zh_en_test_public_release.jsonl
@jayleicn
Hi, thanks for your great project.
The first bug
I just found a bug when using:
bash baselines/mxml/scripts/inference.sh tvr-video_sub-share_vid_enc1_sub_enc1_q_enc1_pred0_q_mappi0_input_p0lw_neighbors1.0_query_sub1_sub2-exp-2021_12_09_17_40_04 test_public --tasks VCMR VR
After comparing with https://github.com/jayleicn/TVRetrieval/tree/master/data, I fixed it by:
from
/mTVRetrieval/data/tvr_zh_en_test_public_release.jsonl
to
/mTVRetrieval/standalone_eval/data_archive/tvr_zh_en_test_public_archive.jsonl
The second bug
And then, I encountered another bug:
2021-12-13 10:55:05.343:INFO:baselines.xml.inference - Loaded model saved at epoch 41 from checkpoint: baselines/mxml/results/tvr-video_sub-share_vid_enc1_sub_enc1_q_enc1_pred0_q_mappi0_input_p0lw_neighbors1.0_query_sub1_sub2-exp-2021_12_09_17_40_04/model.ckpt
2021-12-13 10:55:05.343:INFO:baselines.xml.inference - CUDA enabled.
2021-12-13 10:55:05.349:INFO:__main__ - Starting inference...
2021-12-13 10:55:05.349:INFO:baselines.xml.inference - Computing scores
Computing query2video scores: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:01<00:00, 3.67it/s]
2021-12-13 10:55:07.000:INFO:baselines.xml.inference - Inference with full-script.
Computing q embedding: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:34<00:00, 3.16it/s]
[VR] Loop over queries to generate predictions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:00<00:00, 12867.78it/s]
[VCMR] Loop over queries to generate predictions: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:01<00:00, 4474.36it/s]
2021-12-13 10:55:43.190:INFO:baselines.xml.inference - Saving/Evaluating before nms results
Traceback (most recent call last):
File "baselines/mxml/inference_shared.py", line 57, in <module>
start_inference()
File "baselines/mxml/inference_shared.py", line 51, in start_inference
tasks=opt.tasks, max_after_nms=100)
File "/home/henry/Desktop/mTVRetrieval/baselines/xml/inference.py", line 490, in eval_epoch
use_desc_type=opt.dset_name == "tvr")
File "/home/henry/Desktop/mTVRetrieval/standalone_eval/eval.py", line 268, in eval_retrieval
match_number=match_number, verbose=verbose, use_desc_type=use_desc_type)
File "/home/henry/Desktop/mTVRetrieval/standalone_eval/eval.py", line 148, in eval_by_task_type
desc_types.append(desc_type2idx[gt_item["type"]])
KeyError: 'type'
Similarly, it seems that I fixed it by comparing TVR code and mTVR code.
mTVRetrieval/baselines/xml/inference.py
Lines 486 to 487 in 472e811
from
# if opt.eval_split_name == "val": # since test_public has no GT
if opt.eval_split_name in ["val", "test_public"]: # since test_public has no GT
to
if opt.eval_split_name == "val": # since test_public has no GT
# if opt.eval_split_name in ["val", "test_public"]: # since test_public has no GT
2021-12-13 11:35:34.146:INFO:baselines.xml.inference - Loaded model saved at epoch 41 from checkpoint: baselines/mxml/results/tvr-video_sub-share_vid_enc1_sub_enc1_q_enc1_pred0_q_mappi0_input_p0lw_neighbors1.0_query_sub1_sub2-exp-2021_12_09_17_40_04/model.ckpt
2021-12-13 11:35:34.146:INFO:baselines.xml.inference - CUDA enabled.
2021-12-13 11:35:34.153:INFO:__main__ - Starting inference...
2021-12-13 11:35:34.153:INFO:baselines.xml.inference - Computing scores
Computing query2video scores: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:01<00:00, 3.61it/s]
2021-12-13 11:35:35.836:INFO:baselines.xml.inference - Inference with full-script.
Computing q embedding: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:35<00:00, 3.10it/s]
[VR] Loop over queries to generate predictions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:00<00:00, 12695.44it/s]
[VCMR] Loop over queries to generate predictions: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:01<00:00, 4489.10it/s]
2021-12-13 11:36:12.653:INFO:baselines.xml.inference - Saving/Evaluating before nms results
2021-12-13 11:36:19.497:INFO:__main__ - en metrics_no_nms
None
2021-12-13 11:36:19.498:INFO:__main__ - en metrics_nms
None
2021-12-13 11:36:19.498:INFO:baselines.xml.inference - Computing scores
Computing query2video scores: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:02<00:00, 2.89it/s]
2021-12-13 11:36:21.592:INFO:baselines.xml.inference - Inference with full-script.
Computing q embedding: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:35<00:00, 3.11it/s]
[VR] Loop over queries to generate predictions: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:00<00:00, 12988.23it/s]
[VCMR] Loop over queries to generate predictions: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5445/5445 [00:01<00:00, 4037.06it/s]
2021-12-13 11:36:58.450:INFO:baselines.xml.inference - Saving/Evaluating before nms results
2021-12-13 11:37:05.235:INFO:__main__ - zh metrics_no_nms
None
2021-12-13 11:37:05.235:INFO:__main__ - zh metrics_nms
None
But I don't know exactly whether this approach is correct. :)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.