Bad performance of default encoder about vvenc HOT 13 CLOSED

@f11894 nothing wrong in correcting errors 😄
Yes, I didn't specify framerate. For a bit rate, I read the reporting number from the encoder. So my previous comparison to AV1 is invalid)
Will redo that.
@f11894 Thanks)
@adamjw24 My AV1 vs VVenC data is invalid, I will redo that and report back

from vvenc.

There are a few issues there.

So Dota2 is a screen-content sequence. Screen content specific tools are so far not supported in VVenC, since we concentrated on natural content so far. So stuff like IBC and TS will be coming in some future update. This will make a massive difference.
I'm curious how the test would behave with for natural content.

Further, currently anything above 4 threads for 1080p is basically burning resources, so stick to that and maybe run 3 QP points parallel.

Since you're benchmarking agains VMAF, I would suggest to switch off QPA (--qpa 0). Also, did you inspect the sequences visually? With VMAF, for the really high scores (above 90), the visual differences are very small.

Also, why does aomenc start at 1.5kbps and a 90 VMAF score? Will it not go any lower? What command line did you use for aomenc?

from vvenc.

@adamjw24 thanks for your reply.

I will retest it on natural content, I hope park_joy from https://media.xiph.org/video/derf/ would be sufficient.

Further, currently anything above 4 threads for 1080p is basically burning resources, so stick to that and maybe run 3 QP points parallel.

Will try that 👍

Also, did you inspect the sequences visually?
Yes, it looks fine on low qp, but bitrates are high

Since you're benchmarking against VMAF

I benchmark against VMAF, PSNR, SSIM, MS-SSIM. For example how it's done here benchmark repo
All other metrics show the same relations.

With VMAF, for the really high scores (above 90), the visual differences are very small.

They are noticeable, and VMAF 90 is at the level of 'youtube' quality which is fine.
(in my opinion) the quality difference is noticeable for VMAF >80 separated by 3 points, so even for 90+ scores there is still visual gradation of encoded quality.

Also, why do aomenc start at 1.5kbps and a 90 VMAF score

For testing, I was using the encoding ladder of 50-40-30-20 CQ values for aomenc and 40-35-30-25 for vvenc.
Aomenc 50 cq starts at ~1500 Kbps and 40 qp vvenc starts at ~700 Kbps.

What command line did you use for aomenc?

I was using my project Av1an to simplify piping and encoding.
Encoding settings: 10 bit piping, 1 pass
-b 10 --threads=12 --end-usage=q --cq-level=N --cpu-used=N , that's all

Exact scripts used for benchmarking can be found in this repo
This comparison is still WIP. Plot generated with plot_data.py

from vvenc.

New tests on park_joy 50 frames sample return this:

from vvenc.

Thanks for providing the update, this looks more like what we would expect.

We are working on some more extensive tests an comparisons too, so stay tuned.

Since the initial question is resolved, if there is no follow-up question, I will close the issue soon.

from vvenc.

@adamjw24 👍

from vvenc.

@adamjw24
Made benchmark on a full sequence. Please check it, correct me where I'm wrong
https://www.reddit.com/r/AV1/comments/ir6ekz/vvc_vvenc_vs_av1_aomenc_comparison/
https://github.com/master-of-zen/AV1-benchmarks/tree/master/12.09.2020%20VVCENC%20vs%20AV1
🤗

from vvenc.

Me and my colleague had a look at your results and tried cross-checking them. We cannot really reproduce the results.

The sequence you're using, park_joy is 50fps, so an intra-period of 32 frames is quite short (that's an RA-entry point every 2/3s). Industry standard would be having an RA-entry point every 1-4 seconds. You should try increasing the IP to 48 frames. We'll try to rewrite the semantics on that parameters to take the IP in time units rather than in frames for the future. At least optionally. For now you need to set this manually to a proper value.

Further, it seems you still have the QPA on, which sometimes has some issues with the quality measures. We are looking into that.

Try the following cmd (for the park_joy sequence):

vvencapp --input input.yuv --size 1920x1080 --rate 50 --intraperiod 48 --format yuv420_10 --output N_QP.vvc --preset N --qp QP --qpa 0 --threads 4

Please run the aomenc with 4 threads also, for fair comparison (I know their encoder scales way better for now, but that not the point in this first release).

from vvenc.

@adamjw24 Thanks 😄.

Is it reasonable to set aomenc max-kf-distance as vvc --intraperiod?

Will rerun.

from vvenc.

@master-of-zen Did you get a chance to have generate new numbers?

from vvenc.

@adamjw24 Sorry, currently busy.
When will run a benchmark of --intraperiod values first. (48, 64, 80, 96 112, 128, 144, 160, 176, 192, 208, 224, 240 .. 480 )

from vvenc.

intraperiod values are all options from 64 to 480 with step 16
Worst 480, best 64.(Sum of all metrics)
Worst 32, best 432 (Only VMAF)
BD-rates:

VVC 480 -> VVC 64 (SUM of all metrics)

Vmaf    BD rate: 0.967 %
PSNR    BD rate: -0.079 %
SSIM    BD rate: -7.591 %
MS-SSIM BD rate: -8.203 %
Time Diff:        +17.1 %

VVC 32 -> VVC 432 (Only VMAF)

Vmaf    BD rate: -5.394 %
PSNR    BD rate: -1.909 %
SSIM    BD rate: 6.1 %
MS-SSIM BD rate: 7.445 %
Time Diff:        +7.5 %

All other periods values are within the margin of error.

The second benchmark AV1 vs VVC will be done with --qpa 0 and --intraperiod 64

from vvenc.

@master-of-zen It may be rude to ask this question, but is the --framerate set to 50?
If not set, VVenC default framerate value is 60, so the bitrate will be calculated at the wrong framerate.

from vvenc.