Hi @loveofguoke @longqian-zju,

Thank you for your interesting work. I have been really excited about it!

When I run your training script, I encounter this error:

ModuleNotFoundError: No module named 'momentor.eval'

Can you help me locate where the momentor.eval folder is?

Thank you so much!

Hi there

Thanks for sharing your work!

I wonder if you can share the pre-trained model and the evaluation codes for downstream tasks to ensure reproducibility.

Thank you.

Hi @longqian-zju ,

Thank you for sharing the data and great work. Requesting to share the code for reproducibility.

Thanks

Hi @longqian-zju , @loveofguoke

I am trying to train the model, however, the dataset before and after conversion has different formats and both the formats seems in-compatible to train the model.

data_0 = json.load(open("Moment-10M_0.json"))
data_0["jS2fE622RMA"]["qa_data"]

[{'id': 'jS2fE622RMA', 'data_type': 'qa_data', 'variables': {'moment': [0.0, 12.4]}, 'conversations': [{'User': 'Please watch the clip of {moment}. What color is the car with a black stripe?', 'Assistant': 'The car with a black stripe is a black car.'}], 'clip_similarity': 0.2374267578125}, {'id': 'jS2fE622RMA', 'data_type': 'qa_data', 'variables': {'moment': [12.4, 14.233333333333333]}, 'conversations': [{'User': 'Take a look at the segment of {moment}. What is the color of the SUV in the parking lot?', 'Assistant': 'The color of the SUV in the parking lot is blue.'}]

The code in the train_momentor.py creates new format, however, the key prototype or backbone do not exist.

qa_types = ['qa_data', 'instance_qa_data', 'cross_segment_qa_data']
packed_instruction_data = []
for video_name in tqdm(instruction_data):
    for key in instruction_data[video_name]:
        for dialogue in instruction_data[video_name][key]:
            text_dialogue = dialogue['dialogue']
            backbone = dialogue['prototype']['backbone']
            if key in qa_types:
                backbone[0]['User'] = re.sub(r'(' + re.escape(sent_tokenize(text_dialogue[0]['User'])[0]) + r') +', '', backbone[0]['User'])
            packed_instruction_data.append({
                'id' : video_name,
                'data_type' : key,
                'moment' : dialogue['prototype']['variables'].get('moment', None),
                'click_position' : dialogue['prototype']['variables'].get('click_position', None),
                'instance_class' : dialogue['prototype']['variables'].get('instance_class', None),
                'SOURCE_CLIP' : dialogue['prototype']['variables'].get('SOURCE_CLIP', None),
                'content' : dialogue['prototype']['variables'].get('content', None),
                'conversations' : backbone,
            })

Lastly, the packed_instruction_data format of data is not being used in the file train_momentor.py. I would appreciate if you help to resolve these issues.

Query 2: Do we use only data from ['qa_data', 'instance_qa_data', 'cross_segment_qa_data'] types to train the model and ignore the others?

Thanks!

hi @loveofguoke @longqian-zju

I have the following queries on training the model.

• Do we train the model with image-text or video-text pairs? If yes, then which datasets are used for the same.

• How to train with GESM data? The converted data do not have conversations key, however, the source code expect this information. Moreover after conversion, the only present keys are id, data_type, data as used in convert_data_gesm.py.

• I assume we first need to train with GESM data and then perform supervised instruction fine-tuning with Moment-10M. Please confirm.

It would be really helpful if you can share the information to train the model along with appropriate dataset format.

Thanks in advance!

Dear Maintainers,

I hope this message finds you well. I am reaching out to inquire about the instance-event metric data used in your project.

Your work has been a great source of inspiration for my own research, and having access to the instance-event metric data would significantly aid in understanding and replicating your results.

Are there any plans to release this data in the near future? I believe it would be an invaluable resource for the community and would greatly appreciate your consideration.

Thank you for your time and efforts on this project.

Best Regards

Great job! This is a huge boost to Video-LLM.
I wonder that when will the data be accessible?

我看可视化的图，学出来的已经很线性了，那为什么不直接采用固定的编码？比如余弦或者线性映射？有没有做消融实验

Hi @longqian-zju, thanks for sharing this work. I was wondering whether you are using the same videos and annotations for Grounded Event-Sequence Modeling and instruction tuning. The paper did not mention what data (and the size) you used for GESM.