zhengyiluo / meva Goto Github PK

Hi, Thanks for the excellent work and code release.

I tried to reproduce MEVA's MPII3D PA-MPJPE performance, but i couldn't.

With my pre-processed MPII3D data file (made from VIBE official code, kocabas et al. 2020 CVPR), I ran your evaluation code.

Below are my settings and little modifications based on your evaluation code.

1. I didn't evaluated about the video segment of which number of frames is less than 90.
2. I modified your overlapped chunk generation function("get_chunk_with_overlap"). Because mpii3d have a video segment of which number of frames is 90, and the function generates indices set twice( [(0, 90), (0, 90)] ) for the video segment. Even though it needs frame indices only for the 0~89( (0,90) ). This bug made error for that video segment and crashes program while error calculating operation. Below is the actual values of variables in the function.

(Pdb) num_frames
90
(Pdb) chunk_idxes
array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
        80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
       [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
        32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
        64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
        80, 81, 82, 83, 84, 85, 86, 87, 88, 89]])
(Pdb) chunck_selects
[(0, 90), (0, 90)]

So i modified function like below.

def get_chunk_with_overlap(num_frames, window_size = 90, overlap = 0):
    assert overlap % 2 == 0
    step = window_size - overlap 
    chunk_idxes = view_as_windows(np.array(range(num_frames)), window_size, step= step)
    chunk_supp = np.linspace(num_frames - window_size, num_frames-1, num = window_size).astype(int)
    chunk_idxes = np.concatenate((chunk_idxes, chunk_supp[None, ]))
    last_chunk = chunk_idxes[-1][:step][-1] - chunk_idxes[-2][:step][-1] + int(overlap/2)
    chunck_selects = get_chunk_selects(chunk_idxes, last_chunk, window_size= window_size, overlap=overlap)
    
    return chunk_idxes, chunck_selects

def get_chunk_with_overlap(num_frames, window_size = 90, overlap = 0):
    assert overlap % 2 == 0
    step = window_size - overlap 
    chunk_idxes = view_as_windows(np.array(range(num_frames)), window_size, step= step)
    chunk_supp = np.linspace(num_frames - window_size, num_frames-1, num = window_size).astype(int)
    
    if num_frames != 90:
        chunk_idxes = np.concatenate((chunk_idxes, chunk_supp[None, ]))
        last_chunk = chunk_idxes[-1][:step][-1] - chunk_idxes[-2][:step][-1] + int(overlap/2)
        chunck_selects = get_chunk_selects(chunk_idxes, last_chunk, window_size= window_size, overlap=overlap)
    else:
        chunck_selects = [(0, 90)]
    return chunk_idxes, chunck_selects

After that, evaluation code worked, but result is pretty low than your result in this repo.

'mpjpe': 99.91301986073121,
'mpjpe_pa': 66.27225724514501,
'vel_err': 15.167348631137918,
'acc_err': 14.774026845370512,
'auc': 0.6680740727485766,
'pck': 94.77570523202951

Would you advice for the evaluation on mpii3d for the reproduction? Thanks!

Hello and thanks for the code! I am trying to run the sample code of the algorithm, but I find problems at the inference stage. I can't track it because it doesn't throw me any associated errors.

(env) laila@laila:~/MEVA$ python scripts/run_meva_on_video.py --cfg train_meva_2 --vid_file zen_talking_phone.mp4 --output_folder results/output --exp train_meva_2
/home/laila/Nexar/MEVA/meva/utils/transformations.py:1927: UserWarning: No module named 'meva.utils._transformations'
warnings.warn(str(err))
Running "ffmpeg -i zen_talking_phone.mp4 -f image2 -v error /tmp/zen_talking_phone_mp4/%06d.png"
Images saved to "/tmp/zen_talking_phone_mp4"
Input video number of frames 900
Running Multi-Person-Tracker
100%|███████████████████████████████████████████| 75/75 [00:44<00:00, 1.70it/s]
Finished. Detection + Tracking FPS 20.40
loading VAErecV2 model from checkpoint: results/meva/vae_rec_2/models/model_1000.p
Loaded pretrained weights from "results/meva/train_meva_2/model_best.pth.tar"
Running MEVA on each tracklet...
0%| | 0/1 [00:00<?, ?it/s]Terminado (killed)

If anyone has any idea why this error is due, or how to show in more detail that it does not work, I would be grateful!

hello, i was impressed with your sota achivement, but when i tried the demo code, the result seems like not fit the person, am i got mistakes somewhere?

Hi when I run the command python scripts/run_meva_on_video.py --cfg train_meva_2 --vid_file zen_talking_phone.mp4 --output_folder results/output --exp train_meva_2 I get the error below:

/home/melih/Desktop/virtualTry/MEVA/meva/utils/transformations.py:1927: UserWarning: No module named 'meva.utils._transformations'
warnings.warn(str(err))
Running "ffmpeg -i zen_talking_phone.mp4 -f image2 -v error /tmp/zen_talking_phone_mp4/%06d.png"
Images saved to "/tmp/zen_talking_phone_mp4"
Input video number of frames 901
Running Multi-Person-Tracker
100%|███████████████████████████████████████████| 76/76 [01:06<00:00, 1.15it/s]
Finished. Detection + Tracking FPS 13.63
loading VAErecV2 model from checkpoint: results/meva/vae_rec_2/models/model_1000.p
Loaded pretrained weights from "results/meva/train_meva_2/model_best.pth.tar"
Running MEVA on each tracklet...
100%|█████████████████████████████████████████████| 1/1 [00:53<00:00, 54.00s/it]
VIBE FPS: 16.69
Total time spent: 125.86 seconds (including model loading time).
Total FPS (including model loading time): 7.16.
Saving output results to "results/output/zen_talking_phone/vibe_output.pkl".
Rendering output video, writing frames to /tmp/zen_talking_phone_mp4_output
0%| | 0/901 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/latebind.py", line 41, in call
return self._finalCall( *args, **named )
TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "scripts/run_meva_on_video.py", line 334, in
main(args)
File "scripts/run_meva_on_video.py", line 260, in main
mesh_filename=mesh_filename,
File "/home/melih/Desktop/virtualTry/MEVA/meva/utils/renderer.py", line 105, in render
rgb, _ = self.renderer.render(self.scene, flags=render_flags)
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/pyrender/offscreen.py", line 102, in render
retval = self._renderer.render(scene, flags, seg_node_map)
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/pyrender/renderer.py", line 125, in render
self._update_context(scene, flags)
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/pyrender/renderer.py", line 737, in _update_context
p._add_to_context()
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/pyrender/primitive.py", line 324, in _add_to_context
self._vaid = glGenVertexArrays(1)
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/latebind.py", line 45, in call
return self._finalCall( *args, **named )
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/wrapper.py", line 657, in wrapperCall
result = wrappedOperation( *cArguments )
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 401, in call
if self.load():
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 390, in load
error_checker = self.error_checker,
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 148, in constructFunction
if (not is_core) and not self.checkExtension( extension ):
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/platform/baseplatform.py", line 270, in checkExtension
result = extensions.ExtensionQuerier.hasExtension( name )
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/extensions.py", line 98, in hasExtension
result = registered( specifier )
File "/home/melih/anaconda3/envs/pose/lib/python3.6/site-packages/OpenGL/extensions.py", line 105, in call
if not specifier.startswith( self.prefix ):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

The original VIBE does not use Human3.6M dataset, so could you please provide the h36m_test_db.pt file? Thank you in advance!

hello, good job, and when will the train code come out

hi great project!

When I run GetStart command,
the error called:
ImportError: cannot import name 'ModelOutput' from 'smplx.body_models'
and
No matching distribution found for multi_person_tracker

Hi! Thanks for your working!

Here I find some wrong function which do not have any definition, I can't find these definitions elsewhere either.
such as get_ordered_action_names, interpolated_traj_new and cam_mapping
cause I need to preprocess the Human3.6m datasets with this file.

I would be verrrrrrrrrrry appreciate it if you could improve it. Thank you so much!

Similar to #7, when I run prepare_data.sh the downloaded zip file does not contain meva_data/yolov3.weights or meva_data/model_best.pth.tar which are both referenced in the script and needed to run the demo. Could you add these files to the drive link? Thanks!

In the code I see that you extract all the frames of the video and then ran the tracker and inference on the code. My question is that is there a possibility of running this in real time on an a live video stream if I start modifying the code or does your model requires complete video? Also are there any FPS benchmarks for this model on GPUs? Thanks

Hello, the prepare_data.sh file was recently updated due to an expired link. However, the folder that is now downloaded does not contain the files that the remainder of the script references, specifically model_best.pth.tar and yolov3.weights. This of course causes issues when trying to use the run_meva_on_video.py script.

Can you confirm that this issue or update the google drive link to a folder that contains the appropriate files?

Hello

Thanks for the great work.
I have a general question, I like to learn how your code is generally working without downloading the datasets, if I want just give a random input/output to the model and do the training and testing phase just to get familiar with the code what shape should the input and gt be?
would it be possible for you to just provide a toy sample please?