lukasbommes / mv-extractor Goto Github PK

Great work first, much thanks to the author, saving a lot of time coding in c.

But I am not really sure about the mechanism here. And I do not know if the function of this project is working as expected. For example, my purpose is to use this motion vector to partially replace the optical flow process in video processing. And I generate my video using ffmpeg with exactly the following command:

ffmpeg -r 60 -f image2 -s 1280x720 -i %08d.png -bf 0 -refs 1 -vcodec libx264 -crf 25 -pix_fmt yuv420p test.mp4ffmpeg -r 60 -f image2 -s 1280x720 -i %08d.png -bf 0 -refs 1 -vcodec libx264 -crf 25 -pix_fmt yuv420p test.mp4

Two settings are worth noting here, I have removed the B frames and set the reference frame number of P frame to only 1, so basically this setting is very close to optical computation (each frame is only referenced from the previous frame).

However, I found that the generated video uses both inter and intra macroblocks for compression. Say, in a frame, two macroblocks are numerically identical. It is reasonable to use one to reference another, leaving only a motion vector to save. But this motion vector does break the common understanding of "motion" as this process has nothing to do with temporal movement. It is simply a strategy to save some memory within one frame.

I want to confirm with the author if this situation is considered when extracting motion vectors. Any suggestion, please shoot me.

Thank you for developing this great tool for motion extraction.

I tried to use the docker you provided, but the codes reported an error. The error message is "No module named 'mvextractor'". Looking forward to your reply。

Hi,
I'm trying your project mv-extractor to get timestamp from the camera.
While in 30 FPS it works very nice, on 2 FPS I have a delay of about 3-4 seconds between the time that the camera prints on the frame to the timestamp that returns from the mv-extractor.
is it possible to get high accuracy in low rates or it is only working on high rates?
I added an image from a camera I have: the white text on the image is the time that the camera prints on the frame. the blue text is the timestamp that arrived from the camera. you can see a time diff of 4 seconds
thanks.

Hello. I'm thinking about trying to use this in a machine learning project, and I'm wondering if I'd be able to extract all this information, do some processing on it, and then combine it back into video? As far as I know there's currently no way to do this

how much free space is required for the installation? it take considerable amount of time to install

Hello,

I am using this package in a project to synchronize multiple different camera streams, which I then need to process. In this processing, I have packages that require opencv versions >= 4.6, which are in conflict with the requirements of this package. Would it be possible to update the package dependencies to allow using newer opencv versions?

Thank you very much for the great work either way!

Hello,

Thanks for sharing the code!
I was able to run it effortlessly on my own machine. However, I am having trouble running this in the headless mode on a remote server without a graphics card. This is the error:
'''
Sucessfully opened video file
Frame: 0 timestamp: 1621037937.914803 | frame type: I | frame size: (720, 1280, 3) | motion vectors: (0, 10) | elapsed time: 0.12874823901802301 s
Unable to init server: Could not connect: Connection refused

(Frame:139): Gtk-WARNING **: 00:18:57.922: cannot open display: localhost:10.0
root@5a0189971e6a:/home/video_cap#
'''

In a separate window, I got this error.
(base) user@srv~$ xhost +
xhost: unable to open display "localhost:11.0"

Thanks for any help.

Does this tool decode the video or does it retrieve the motion vectors without decoding it? I am looking for a solution with low CPU usage. I looked at the code and it seems the entire frame is decoded, however, I am not a C++ expert. Thank you.

Thanks for your work, I have a frame-type question. I found out the frame-type of the first frame from an h.264 video which is shown in the mv extractor is not an I frame (is a P frame).

The source parameter in the numpy array "motion_vectors" is always -1 or 1. I need the exact reference frame number used by the macroblock (Not just whether it used past/future frame). Could you please help me with that?

Dear all,
Thanks for your interesting implementation. There is a question, Is the motion vector extracted after the decoding process?
Best regards

Hi @LukasBommes @sfmt-auto

Thanks for you repo, this is very impressive. I'm using another repo https://github.com/LukasBommes/rtsp-streamsync/tree/master which is rely on this repo and that helps me to get synchronized frames from ip camera stream.

I'm trying to build a docker image which is based on ubuntu 20.04. So, I changed the Dockerfile of rtsp-streamsync repo

RUN cd $HOME && \
  git clone -b v0.0.0 https://github.com/LukasBommes/mv-extractor.git video_cap && \
  cd video_cap && \
  chmod +x install.sh && \
  ./install.sh

to

RUN cd $HOME && \
    git clone -b master https://github.com/LukasBommes/mv-extractor.git video_cap && \
    cd video_cap && \
    chmod +x install_ffmpeg.sh && \
    chmod +x install_opencv.sh && \
    ./install_ffmpeg.sh && \
    ./install_opencv.sh

cuz I wanna try if the docker image based on ubuntu 22.04 can be directly used on 20.04

but i got this error:

I'm wondering does this repo provide docker image based on ubuntu 20.04 or do you have any suggestions on why is this error?

Any comments will be helpful!

Thank you!

Thank you so much for open-sourcing your code! But I used the provided environment to print the motion vector directly and found that it is not complete, (There is no corresponding motion vector for some areas). Looking forward to your reply!

VideoCap::open() sets the number of threads to whatever is returned by std::thread::hardware_concurrency() in this line. However, on machines with many cores the number of threads will exceed the maximum recommended by FFmpeg (which is 16 as defined here). This may degrade performance, and FFmpeg will issue an annoying warning says "Using a thread count greater than 16 is not recommended". It may be wiser to follow the recommendation and limit the number of threads to no larger than 16.

pip install motion-vector-extractor
ERROR: Could not find a version that satisfies the requirement motion-vector-extractor (from versions: none)
ERROR: No matching distribution found for motion-vector-extractor

Hi, first of all thanks for your library, it's very helpful. I'm wondering how to get only I - frames or decompress P/B via I -frames. Can you help me please?

Hi, thanks for your excellent work.

I found that when I used this tool to extract my video（h264 codec）with fps=50(2000frame), the result shows there is only half of num of frames(1000 frames). I am wondering Is there any framerate threohld setting in your video-reading module?

Thanks

Seek help. I have installed opencv and ffmpeg source code, but encountered the following problems when run "python -m pip install .":

src/mvextractor/video_cap.cpp:218:41: error: ‘AVPacket’ {aka ‘struct AVPacket’} has no member named ‘synced’
if (this->is_rtsp && packet.synced) {
^~~~~~
src/mvextractor/video_cap.cpp:222:32: error: ‘AVPacket’ {aka ‘struct AVPacket’} has no member named ‘last_rtcp_ntp_time’
ntp2tv(&packet.last_rtcp_ntp_time, &tv);
^~~~~~~~~~~~~~~~~~
src/mvextractor/video_cap.cpp:223:51: error: ‘AVPacket’ {aka ‘struct AVPacket’} has no member named ‘timestamp’
double rtp_diff = (double)(packet.timestamp - packet.last_rtcp_timestamp) / 90000.0;
^~~~~~~~~
src/mvextractor/video_cap.cpp:223:70: error: ‘AVPacket’ {aka ‘struct AVPacket’} has no member named ‘last_rtcp_timestamp’
double rtp_diff = (double)(packet.timestamp - packet.last_rtcp_timestamp) / 90000.0;

I check the AVPacket struct of the "ffmpeg_build/include/libavcodec/avcodec.h" file, and found they really don't exist either in the new version (ffmpeg 5.0, ffmpeg 4.1) or the old version ( ffmpeg 3.4).

Dear author and other people who are interested,

I am interested in ML on compressed video recently, however, after trying few codes on this repo, I have the following concerns.

1. why the motion vector is stored by int64 by default? from my experiment, the size of it can easily be larger than the original video.
2. even if I change the type of motion vector to uint8, the size of it is still 4* larger than the original video. I guess there is further compression behind H.264 (I am new to video codec), can anyone confirm it?
3. read the pre-saved .npy motion vector seems only have a very limited advantage compared to directly reading the RGB. (7s for motion vector, 8s for OpenCV RGB reading, 30minutes video). I understand reading the .npy is not the same as reading the byte from video by C++ (although np.load is C++ backend), but since the decoder is not so heavy, I still feel that only reading motion vector can give limited benefit.
4. Besides, can I use mv-extractor to directly get the motion vector?

I am appreciate to any comments or help! thanks in advance.

I try to add mv_extractor with setup.py

The module is installed but this problem persist, I only need to import mv_extractor to python. Using Ubunto 20.04

Hi! This tool is awesome and easy to use, for a project I am trying to extract the --dump motion vector values and turn them into polar coordinates for instancing in another project. It seems that for every frame though, I receive a different number of motion vectors. Is this normal?
below is me unsorted, printing the number of lines in each motion vector .npy file. I am working with a 720p video and am expecting 3600 motion vectors based on the 16 x 16 kernel

Could this library be used in Windows?
I tried to compile it - modified setup.py - but still I am having some issues with ffmpeg and with different C++ compiler errors.

Hi, Thank you for developing this great tool for motion extraction.

I used the tool to extract the motion vectors in the example video. Let's say we have the reference frame, the current frame, and the motion vectors corresponding to current frame.

I used the motion vectors to apply motion compensation on the reference frame in order to reconstruct the current frame. After motion compensation, there are some empty macro blocks left in the frame. They are the macro blocks which no motion vector was pointed to in the current frame. I suppose that for these macro blocks, no similar macro block in the reference frame was found.

Now I have two questions. Is there anywhere in the code that I can set the threshold for finding the similar Macroblock. I kind of hope to have less empty Macroblocks in the motion compensated frames. My second question is that is it possible that we have bi-directional frames using this tool? I think if we have bi-directional frames, there would be less empty macrobloacks in the motion-compensated frame.

I appreciate your response. Thank you!

Reference frame:

Current frame:

motion-compensated reference frame:

Hi @LukasBommes,

Thank you very much for your excellent work. I was able to compile this into a conda package here: https://anaconda.org/necla-ml/mv-extractor which can be installed with conda install -c necla-ml mv-extractor.

However, when ever I use mvextractor alongside pyav (conda install av -c conda-forge). The extractor gets stuck at this line: https://github.com/LukasBommes/mv-extractor/blob/master/src/mvextractor/video_cap.cpp#L199. Can you please let me know if you have any idea about why this could be happening. I am guessing this is caused because pyav also creates a wrapper around the FFMPEG library which could be conflicting with the mvextractor somehow. Here is a small script to reproduce the bug:

import av
# if we remove the av import it works fine 

def extract_motion_vectors(video_pth):
    from mvextractor.videocap import VideoCap

    cap = VideoCap()
    # open the video file
    ret = cap.open(video_pth)

    idx = 0
    while ret:
        # read next video frame and corresponding motion vectors
        ret, frame, motion_vectors, frame_type, timestamp = cap.read()
        if not ret:
            break
        print(frame.shape)
        idx += 1
    
    cap.release()

if __name__ == '__main__':
    outputs = extract_motion_vectors('<path to video file>')
    # import av
    # it works fine if av is imported after the extraction

thanks for your responses, I would like your help to use your module inside my project - I only need the videoCap API for python in Pycharm.

is there a guide for using the docker inside Pycharm? or the module itself without the docker?

I was looking to implement several types of motion detection with a webcam on an embedded system. However, for this project to work I'd either need to implement all of this myself or have direct access to the webcam via the VideoCap-class, just like OpenCV does it using the device index. Would this be possible to implement?

When I execute this command I got the following error:
sudo docker build . --tag=mv_extractor

But this folder exists, I don't know how to solve this problem, please help!