This project is based on the paper Making a long story short: A multi-importance fast-forwarding egocentric videos with the emphasis on relevant objects on the Special Issue on Egocentric Vision and Lifelogging Tools at Journal of Visual Communication and Image Representation (JVCI 2018). It implements a fast-forward method for first-person videos based on multi-importance approach.

For more information and visual results, please acess the project page.

• Michel Melo da Silva - PhD student - UFMG - [email protected]
• Washington Luis de Souza Ramos - PhD student - UFMG - [email protected]
• Felipe Cadar Chamone - Undergraduate Student - UFMG - [email protected]
• João Pedro Klock Ferreira - Undergraduate Student - UFMG - [email protected]
• Mario Fernando Montenegro Campos - Advisor - UFMG - [email protected]
• Erickson Rangel do Nascimento - Advisor - UFMG - [email protected]

Federal University of Minas Gerais (UFMG)
Computer Science Department
Belo Horizonte - Minas Gerais - Brazil

VeRLab: Vision and Robotic Laboratory
http://www.verlab.dcc.ufmg.br

• MATLAB 2015a
• OpenCV 2.4 (Tested with 2.4.9 and 2.4.13)
• Armadillo 6 (Tested with 6.600.5 -- Catabolic Amalgamator)
• Boost 1 (Tested with 1.54.0 and 1.58.0)
• Doxygen 1 (for documentation only - Tested with 1.8.12)

Just follow the steps in PythonScripts folder.

• The following flowchart depicts the processing steps of our method:

1. Optical Flow Estimator:

   The first step processing is to estimate the Optical Flow (OF) of the input video.

   1. The folder Vid2OpticalFlowCSV contains the modified Poleg et al. 2014 Flow Estimator code from the link to run in the Linux system.
   2. Navigate to the folder compile the code.
   3. Into the Vid2OpticalFlowCSV folder, run the command:

./optflow -v < video_filename > -c < config.xml > -o < output_filename.csv >

Save the output file using the same name of the using with extension .csv.

2. Semantic Extractor:

   The second step is to extract the semantic information over all frames of the Input video and save it in a CSV file. On MATLAB console, go to the project folder and run the command:

>> ExtractAndSave(< Video_filename >, < Semantic_extractor_name >)

3. Create Experiment

   To run the code, you should create an experiment entry. On a text editor, add a case to the GetVideoDetails function in the file SemanticSequenceLibrary.m:

function [videoFile, startInd, endInd, filename, fps] = GetVideoDetails(video_dir,exp_name)

 ...

 case < Experiment_name >
     filename = < video_filename >;
     startInd = < start_index_frame > ;
     endInd   = < final_index_frame >;
     fps      = < video_frames_per_second >;
					
  ... 
  

4. Semantic Fast-Forward

   After the previous steps, you are ready to accelerate the Input Video. On MATLAB console, go to the project folder and run the command:

>> SpeedupVideo(< Video_dir >, < Experiment_name >, < Semantic_extractor_name >, ['Speedup', < rate >] )

5. Configure Video Parameters

   After the Semantic Fast-Forward step, the accelerated video is create. Now we are going to stabilize the output video. The first stabilization step is to configure the video parameters in the file acceleratedVideoStabilizer/experiment.xml. Follow the instructions described into the file.

6. Accelerate Video Stabilizer

   Navigate to the <project_folder>/acceleratedVideoStabilizer/ folder. Follow the instructions described into the <project_folder>/acceleratedVideoStabilizer/README.md file to compile and run the code.

   The output of this step is the stabilized semantic fast-forward video.

1. EgoSampling: Y. Poleg, T. Halperin, C. Arora, S. Peleg, Egosampling: Fast-forward and stereo for egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4768–4776. doi:10.1109/CVPR.2015.7299109.
2. LK_Blocked_Optical_Flow: Y. Poleg, C. Arora, S. Peleg, Temporal segmentation of egocentric videos, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2537–2544. doi:10.1109/CVPR.2014.325.
3. NPD_Face_Detector: S. Liao, A. K. Jain, S. Z. Li, A fast and accurate unconstrained face detector, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2) (2016) 211–223. doi:10.1109/TPAMI.2015.2448075.
4. PMT_Pedestrian_Detector: P. Dollar, Piotr’s Computer Vision Matlab Toolbox (PMT), https://github.com/pdollar/toolbox.

If you are using it to academic purpose, please cite:

M. M. Silva, W. L. S. Ramos, F.C. Chamone, J. P. K. Ferreira, M. F. M. Campos, E. R. Nascimento, Making a long story short: A multi-importance fast-forwarding egocentric videos with the emphasis on relevant objects, in: Special Issue on Egocentric Vision and Lifelogging Tools at Journal of Visual Communication and Image Representation. doi:10.1016/j.jvcir.2018.02.013.

@article{Silva2018, title = {Making a long story short: A Multi-Importance fast-forwarding egocentric videos with the emphasis on relevant objects},
author = {Michel M. Silva and Washington L. S. Ramos and Felipe C. Chamone and João P. K. Ferreira and Mario F. M. Campos and Erickson R. Nascimento},
journal = {Journal of Visual Communication and Image Representation},
volume = {53},
number = {},
pages = {55 – 64},
year = {2018},
issn = {1047-3203},
doi = {10.1016/j.jvcir.2018.02.013}
}