avrateNG is a video, image, and general multimedia rating system, based on a simple web interface. avrateNG is developed as part of research in the AVT Group.

avrateNG was inspired by AVrate.

If you want to implement an online test, please checkout AVrate Voyager

• Operating system: Windows, Linux or macOS
• Python 3
  • Windows: you can use the provided Python 3 distribution and setup files in thirdparty folder
  • Linux: sudo apt-get install python3
  • macOS: Use Homebrew and run brew install python

Furthermore you also need a player, e.g. for Linux (Ubuntu) or macOS you can use mpv (install it via sudo apt-get install mpv or brew install mpv); for Windows you can use the version that is stored in thirdparty.

For the generate_plots.py script you need pandas and seaborn:

pip3 install pandas seaborn

Note: generate_plots.py is experimental.

Before you should start with your specific processed video files, you should try to run avrateNG. If you correctly checkout the repository, everything should work.

Just start avrateNG.py and open http://127.0.0.1:12347/ (preferred browser is Chrome/Chromium, it should work also with Firefox and Edge) in your favorite web browser:

./avrateNG.py

The default credentials are:

• User: max
• Password: 123

Change these in the config.json file.

All ratings are stored in a sqlite3 database. For a simple conversion you can use convert_ratings_to_csv.py. This script will create a csv file of all stored ratings and other data. It requires pandas to be installed.

All general settings can be changed in config.json, e.g.

{
    "player": "thirdparty\\mpv-x86_64-20180429-git-dc16d85\\mpv.exe --fs \"{filename}\"", // default Windows player path, \"{filename}\" is a template for the video filename
    "player_linux": "mpv --fs '{filename}'",                                  // Linux player
    "http_user_name": "max",                                                  // user login name
    "http_user_password": "123",                                              // user password
    "http_port" : "12347",                                                    // http port where the service is running
    "rating_template" : "radio1.tpl",                                         // template that will be used , e.g. change it to "radio1.tpl"
    "playlist" : "playlist.list",                                             // Playlist file that will be used (Define your playlist here)
    "template_folder" : "templates",                                          // Folder with your custom templates (Take a look in "/templates/")
    "training" : true,                                                        //  Training stage up front? (true/false)
    "trainingsplaylist" : "training.list",                                    // if training is true, this trainingsplaylist will be presented
    "voiceRecognition"  : false,                                              // rating using voice recognition? (a little buggy...)
    "voiceRecognition_template" : "radio_voice-recognition.tpl",              // template that will be used for voice recognition
    "shuffle": true                                                           // Randomized playback of videos in playlist? (true: Randomization, false: linear playback according to playlist)
}

You just need to change the player or player_linux value in the config.json to your favorite video player corresponding to your operating system, e.g. it also works with Media Player Classic, VLC or ffplay.

Please try use command line flags and no manually configured GUI settings, so that your experiment can be run without spending hours in configuration of the player.

• mpv: some problems with 4K content and 60 fps, and vp9
  • command line arguments: -cache 8388608 -fs --cursor-autohide=0 --osc=no --no-input-default-bindings --hwdec=auto --ontop
• Media Player Classic: problems with 4K, 60fps and vp9
• ffplay: slower than mpv for 4K
• VLC: slowest player ever (not yet tested with version 3.0)
• OptiPlay: the following command works:

thirdparty\OptiPlay-0.7beta1.exe -an -f v210 -video_size 3840x2160 -framerate 60 -i {filename}

The playlist playlist.list consists just of lines with corresponding video files, e.g.

./videos/01.mkv
./videos/02.mkv

You can also define a training playlist training.list.

The playlists to render are defined in the config.json file. Also set training to true or false in there.

You can specify multiple videos in one playlist entry if you separate them by | (spaces are important).

./videos/01.mkv | ./videos/01.mkv
./videos/02.mkv

The configured player will now get as {filename} = "./videos/01.mkv" "./videos/01.mkv".

A 2 second gray video will be played before and after a video, for disabling just remove "gray_video": "..." in your config file.

Just run avrateNG.py -h and you will get the following screen:

usage: avrateNG.py [-h] [-configfilename CONFIGFILENAME] [--standalone]

avrate++

optional arguments:
  -h, --help            show this help message and exit
  -configfilename CONFIGFILENAME
  --standalone          run as standalone version (default: False)

stg7 2019

There are several rating templates implemented, they can be changed in the config.json. Default template is a classic ACR rating scheme.

For a first check how the output format could be used in a small analysis, check the jupyter notebook in the analysis folder

• Steve Göring
• Maximilian Schaab
• Serge Molina
• Anton Schubert

• demographics form: John Dumke and Margaret Pinson

If you use this software in your research, please include a link to the repository and cite the following paper:

@inproceedings{goering2021voyager,
  title={AVRate Voyager: an open source online testing platform},
  author={Steve Göring and Rakesh {Rao Ramachandra Rao} and Stephan Fremerey and Alexander Raake},
  year={2021},
  booktitle={2021 IEEE 23st International Workshop on Multimedia Signal Processing (MMSP)},
  pages={1--6},
  organization={IEEE}
}

GNU General Public License v3. See LICENSE file in this repository.