fifonik / ffmetrics Goto Github PK

The app keeps project name until you close the app.
Expected: After a successful "Save As..." window title should change to the new project name.

Loading a pkl-model fails with latest ffmpeg-build

Taken from FFMetrics.log:

libvmaf WARNING could not read model from path: "D:/[...]/FFMetrics/vmaf-models/vmaf_v0.6.1.pkl"
libvmaf WARNING pkl model files have been deprecated, use json
libvmaf ERROR problem loading model file: D:/[...]/FFMetrics/vmaf-models/vmaf_v0.6.1.pkl
[Parsed_libvmaf_3 @ 000000000b1511c0] libvmaf encountered an error, check log for details

I tested 2 recent ffmpeg-build from gyan.dev:

ffmpeg-4.3.1-2021-01-01-full_build.7z
ffmpeg-4.3.1-2020-11-19-full_build.7z

The older one (2020-11-19) works with pkl-models!

The latest one (2021-01-01) now probably uses libvmaf v2, as it got released 2020-12-04 and doesn't support pkl-models anymore!
You can see in the code, that it only tries to load json-models now and it at least throws warnings since v2.1.0!

Unable to use json-models

After putting recent json-models into the models-folder, they cannot be selected in the model-dopdown-menu of "VMAF options" in the GUI.
I also tried to load a json-model by changing my "FFMetrics.conf", but it doesn't use the specified default model name as can be seen from FFMetrics.log.

For long videos, full stream analysis is unrealistic, especially using vmaf, but the longest segment analysis is only 2 minutes in the preset. In most cases, 2 minutes cannot cover multiple scenes in the video, so I need longer segment to analyze.

I request the option to add a custom segment time.

If I have heavy transcoding going on in the background - the app always crashes. Based on the logs, I guess it crashes when it tries to load media info, and ffmpeg communication times out.

Frame statistics may be very interesting in the final report

Now this tool failed to compare raw 264 stream without container. The reason I guess is we should specify resolution, pixel format and other parameters when there is no container.
And also, I didn't get the ffmpeg command when there is an error even I have selected the option.
Third, I can't drag the chart when I zoomed out it but can't find the proper area. It's hard for me to see the ssim results when the result is approximate 100%.

2021-03-07 16:18:13	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:18:15	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:22:00	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:22:00	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:23:41	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:24:59	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:25:06	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:26:30	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:26:30	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:26:38	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:27:10	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:27:45	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:27:59	ERROR: ERROR: Calc background worker finished: 可为空的对象必须具有一个值。
2021-03-07 16:28:36	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。
2021-03-07 16:28:36	WARNING: FF.MediaInfoGet.Finished: FAIL 未将对象引用设置到对象的实例。

For some reason the VMAF score in the file list is missing a decimal delimiter(point/comma), the other scores are OK though. When hovering however, I do see a delimiter. A clew might be that the VMAF uses a comma in the hover tool-tip, while the others use a point. I'm also seeing this in the CSV export.

Could be related: the graphs for VMAF seem to be broken as well, probably due to this decimal delimiter issue:

The other graphs work perfectly as expected.

An important note and a probable cause: I'm not on an EN-US version of windows. I'm in a NL-BE locale, where we use a comma(,) as the decimal delimiter. I'm guessing the software/code somewhere expects the point instead of the system locale value. I don't know what language the app is written in, but you should be able to get the delimiter that is in use from the system.
Edit: I quickly tested this theory by setting my decimal delimiter in windows to point instead of comma and this fixes/hides the issue.

Offtopic: great little app, very useful!

Hello,
Is there a file size limit? Any time I try to add a reference file it freezes.
I've tried a whole assortment without any luck.

Thank you!

I found your great tool today, but can't use it.
The "Start" button is not clickable and that doesn't change, whatever I do.
I've put ffmpeg.exe into the program folder and the progam recognizes that it's there (as seen on the screenshot).
Also I tried this with different reference and distorted files, but "Start" still doens't become clickable.
What might be the issue?

I converted several 4K videos with HandBrake v1.3.3, x.265 --lossless.
When I load them in the app - Media info stays as N/A.
If I start a project analysis with such videos - PSNR, SSIM, and VMAF won't calculate: progress bar will keep rotating in the related cell in the file list, but project analysis will stop (Button Start appears instead of Stop)

Average numbers are quite misleading when it comes to real-world metric comparisons. When you want to keep an eye on the encoding quality - you need to pay attention to the number of "bad frames" (frames with the lowest scores). Collecting lower percentiles would help to understand the quality of the worst frames/sequences.

The list of files seems to be limited to about 12 files. Additionally, the list box cannot be expanded in height.

It would really be helpful if bitrate (and possibly file size) was listed along with the various metrics scores for the files. Seeing that one file scored better isn't that useful in isolation without also knowing the bitrate, since higher bitrate obviously results in higher quality output. In order to judge the tradeoffs in compression you really need to see both parameters, but right now the only way to view bitrate is by hovering over the file, which takes a second to show up, and is a hassle when comparing multiple files.

I tried to run the app on a VirtualBox VM and the start button would never enable. No error conditions or log file.

FFMetrics is very nice，thank your work.

But when I calculate VMAF for a video file named in Chinese, something is wrong. VMAF score can't display correctly while PSNR and SSIM are fine.

In order to troubleshoot the video itself, I rename 称重达拉崩吧.mkv to 1.mkv. And everything is all right.

I'm not sure if it is the filename(maybe encoding or charcater set, like UTF-8 and GBK) problem.

FFMetrics.log shows

2021-02-01 13:01:42 ffmpeg.exe -hide_banner -nostdin -i C:/Users/privacy/称重达拉崩吧.mkv -i C:/Users/privacy/称重达拉崩吧.mp4 -lavfi [0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf='eof_action=endall:n_threads=7:phone_model=0:pool=harmonic_mean:model_path=C:/Green/Official/FFMetrics/vmaf-models/vmaf_v0.6.1.json:log_fmt=json:log_path=C:/Users/privacy/Temp/FFMetrics/称重达拉崩吧.mkv_24955089-9c51-46e6-83da-1826f396bedd.log' -f null -

I just find 绉伴噸杈炬媺宕╁惂.mkv_24955089-9c51-46e6-83da-1826f396bedd.log in .../Temp/FFMetrics/.

By the way, VMAF's gragh is not frame by frame like the other two. It only appears after all the calculations have been completed. Is this normal？

English is not my native language, please excuse my typing errors.

ffmpeg supports hardware decoders and it would be nice to allow such support if compatible hardware is detected
Like HandBreak options:

Since project completion can take quite some time, it would be great to save the results automatically (or even periodically). Otherwise, computer restart can make us lose all that was calculated.

Hi,

Thx a lot for you great software.

In 0.9.0 Beta, i have an error not see in 0.8.7.

I change source and files to compare, click start and all goes in error.
Re starting the software solve the problem
I also see one freeze in VMAF analyse, but look at the log if there is and error, solved by restarting the software

regards,

please find the log

FFMetrics ERRORS.log

If I select one reference video file and two videos to compare to, graph only shows one of the files even though metrics are calculated for all files.

Using latest version: v0.8.1b

"Bad frames" can be defined as the lowest 1st percentile of a selected metric. For example, if the test sequence has 1000 frames and you use VMAF - "bad frames" would be 10 frames with the lowest VMAF score.

Usually, you would want to review the bad frames visually to understand the quality drops.
Based on the review, you can improve the transcoding or enhancement settings.

Saving at least the list of bad frames would allow us to extract them with FFmpeg later.
But, ideally, bad frames themselves should be saved as well (original and distorted) together with the video analysis CSV report.

Metric value in program is in TextBox so it is possible to focus the field and click Ctrl+C to copy value.
However, magnified focusing TextBox if its value is changed.
So the metric values should be changed to TextBlock/Label when calculating metric instead of TextBox.

Hi,
No plan to include downscale/upscale for tested files ?

I set my ref video and a distorted video, start the analysis for full stream, the CPU and GPU kick up, I get the green curve, but no progress bar or red curve, even after 5min. I can't interact with the dialog. I have to kill the app with Task Mgr.

v0.9.6b on W10 v20H2 x64, Ryzen 7, 64GB RAM, Quadro P400

As of now, the current VMAF options written in log, not the one that were used for calculation.

1. Select mode, mean, untick phone
2. Calculate VMAF
3. Change model, select harmonic mean, tick phone
4. Save results
   =>
   Wrong VMAF options will be saved in csv

I just tried out this really handy program for the first time, and it is super easy to use and very nicely presented, thank you!

The only issue I am running into is that the results seem way off. The previous tool I was using https://github.com/gdavila/easyVmaf takes care of frame sync issues and would be nice to incorporate such a thing with this tool.

For example, here is the results from the same reference and test file from FFMetrics:

Vs results from easyVAMF (quickly thrown into a matplotlib plot for compare)

It was a direct encode from the source with no framerate changes, so I am not sure what goes wrong with the comparison, but it seems to be a somewhat common issue.

It would be nice if the taskbar shows that the project analysis is running.
AFAIK it can be done with TaskbarItemInfo.ProgressState

Related to #16

Right now only ffmpeg commands related to metric calculation logged if such option is ticked.

Auxiliary ffmpeg commands such as getting ffmpeg version, getting supported filters, testing supported vmaf model types, getting media info are not logged if the option is ticked (it is possible to log such commands calling program with "-log-level=debug" command line parameter).

Users should be able to have auxiliary ffmpeg commands logged as well.

So 2 possibilities:

1. If the existing option ticked, add auxiliary ffmpeg commands into log as well
2. Add another options in UI for auxiliary ffmpeg commands

For example, while analyzing big files message "Frame: xxxxxx" is trimmed and only 5 first digits of frame number is displayed.

is it possible to add ffmetric and/or ffmpeg version to the csv files ?

regards,

On multi-CPU configurations PSNR/SSIM can only utilize a few cores so CPU is under-loading and so a few ffmpegs could be executed simultaneously to speedup overall process.

Hi.
Can you implement downscaling/upscaling reference file to specified resolution?
It is impossible to compare reference file 2k to FHD output files without encoding reference file.
It's possible that you can change resolution on the fly with proper filter.
I tested this command but without success.

ffmpeg -i final.mp4 -i original.mp4 -filter_complex "[1:v]scale=1920:1080[final];[final][0:v]libvmaf" -f null -

ut model at /usr/local/share/model/vmaf_v0.6.1.pkl cannot be read successfully.
Caught VmafException: Error loading model (.pkl): Trouble reading the file:/usr/local/share/model/vmaf_v0.6.1.pkl
[Parsed_libvmaf_1 @ 00000238ea9f4e80] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!

Therefore I will not provide a ready-made solution.

• To rid off Newtonsoft.Json dependency
• To use faster graphs rendered

I test with some ffmpeg form Btbn and gyan(4.4, 4.3, lastest nightly, 20201130 nightly), 0.9.4b can use vmaf, 0.9.5b can't.

I'm very confused!

I tested your latest release (v0.8.7) and noticed the errors, when trying to compare 720p and 480p files to a 1080p ref.
After checking the log and also the closed issues #5 and #6, it's clear that FFMetrics still does not support this basic use-case, which is probably super-common!

Why is that? Are you planning to add some sort of auto-scaling feature to deal with different frame sizes in the future?
At least you could show a message to the user, instead of them having to analyze the log for what happened.
The "Media Info" clearly shows the differing resolutions, after files have been added, but it then just let's you continue and fail.

In #5 you state, that up-/downscaling of the test videos is out of the scope of this project, yet in #6 you mention not being aware at first, that ffmpeg doesn't (up-)scale automatically, while using libvmaf and also you would find it useful to add this feature in that case.

Well, it's not that hard to create an ffmpeg-command with a filter-graph, that upscales the test videos to ref size on-the-fly.
See for example here.

Also imho the test videos would always be smaller, when they differ from ref, and should be upscaled for correct results, as Netflix states in their blog entry:
https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12

The upscaling-method should simulate the scaling, that occurs during playback of a low-res file, but usually "bicubic" can be used.
So it would be great, if FFMetrics could also provide an interface/option for that, but it's optional.

Again, I think, that comparing files with smaller frame sizes to the ref is a really important use-case and Netflix also mentions, that it's better to lower the resolution (and also maybe the bitrate), instead of just lowering bitrate alone while keeping the same resolution to achieve better compression results!

Well, I could manually use ffmpeg to give me the vmaf-results for "my" use-case, but I liked the idea of a quick-to-use GUI-frontend that also shows all the metrics as graphs right away. Yet, in its current state FFmetrics is practically useless (at least for me)!

I tried altering the filter via "FFMetrics.conf", but I cannot use "json-models" for recent ffmpeg-build.

{
	"Metrics": {
		"VMAF": {
			"LavfiTemplate": "[0:v]setpts=PTS-STARTPTS,scale=2336:1080:flags=bicubic[main];[1:v]setpts=PTS-STARTPTS[ref];[main][ref]libvmaf='eof_action=endall:psnr=1:ssim=1:ms_ssim=1:phone_model={{phone_model}}:pool={{pool}}:model_path={{model_path}}:log_fmt={{log_fmt}}:log_path={{log_path}}'",
			"Models": {
				"vmaf_4k_v0.6.1.pkl": {
					"height-min": 1081
				},
				"vmaf_v0.6.1.json": {
					"default": true
				}
			}
		}
	}
}

Also the program config gets restored, but "FFMetrics.exe.config" is not changed and there is nothing in AppData or registry either!
Where is the configuration stored?

Its used to be work good, after I update to newest version it always fail to calculate vmaf, but SSIM & PSNR is fine.

here's the log

2021-03-21 05:52:51 ffmpeg.exe -hide_banner -nostdin -i C:/Tools/x265-Yuuki/1.mp4 -i C:/Tools/x265-Yuuki/abs-189-C.mp4_1.mp4 -lavfi [0:v]settb=AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=AVTB,setpts=PTS-STARTPTS[ref];[main][ref]libvmaf='eof_action=endall:n_threads=23:phone_model=0:pool=mean:model_path=C:/Tools/FFMetrics/vmaf-models/vmaf_v0.6.1.pkl:log_fmt=json:log_path=C:/Users/nanoh/AppData/Local/Temp/FFMetrics/d680ea91-61c7-4703-a264-bce065df5fc7.log' -f null -
2021-03-21 05:52:51 ERROR: C:\Tools\x265-Yuuki\1.mp4: Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:/Tools/x265-Yuuki/1.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isom
creation_time : 2021-03-21T09:00:02.000000Z
Duration: 00:03:20.23, start: 0.000000, bitrate: 1187 kb/s
Stream #0:0(und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv), 720x404, 1185 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 29.97 tbc (default)
Metadata:
creation_time : 2021-03-21T09:00:02.000000Z
handler_name : L-SMASH Video Media Handler
encoder : HEVC Coding
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'C:/Tools/x265-Yuuki/abs-189-C.mp4_1.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isom
creation_time : 2021-03-21T09:04:11.000000Z
Duration: 00:03:20.23, start: 0.000000, bitrate: 7605 kb/s
Stream #1:0(und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv), 720x404 [SAR 1:1 DAR 180:101], 7602 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 29.97 tbc (default)
Metadata:
creation_time : 2021-03-21T09:04:11.000000Z
handler_name : L-SMASH Video Media Handler
encoder : HEVC Coding
Stream mapping:
Stream #0:0 (hevc) -> settb
Stream #1:0 (hevc) -> settb
libvmaf -> Stream #0:0 (wrapped_avframe)
libvmaf INFO compute_vmaf() is deprecated and will be removed in a future libvmaf version
Output #0, null, to 'pipe:':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isom
encoder : Lavf58.45.100
Stream #0:0: Video: wrapped_avframe, yuv420p10le, 720x404, q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn, 29.97 tbc (default)
Metadata:
encoder : Lavc58.91.100 wrapped_avframe
libvmaf WARNING could not read model from path: "C:/Tools/FFMetrics/vmaf-models/vmaf_v0.6.1.pkl"
libvmaf WARNING pkl model files have been deprecated, use json
libvmaf ERROR problem loading model file: C:/Tools/FFMetrics/vmaf-models/vmaf_v0.6.1.pkl
[Parsed_libvmaf_4 @ 0000027ca20a2bc0] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0
Conversion failed!

OS: Win 10 19042.610
FFMPEG ver.: 4.3.1