QOpenGLWidget Example? about qtavplayer HOT 20 CLOSED

About opengl and windows. In qt6 by default d3d11 is used. And opengl is not used for rendering.
I tried to make a converting to gl textures like in this draft: https://github.com/valbok/QtAVPlayer/blob/d3d11_gl/src/QtAVPlayer/qavhwdevice_d3d11.cpp
but by default it is useless since we need to know how to use d3d11 textures in QRHI. Which as I can see is not yet supported for qml and widgets. You would need to create your own renderers to support it.

from qtavplayer.

I already made a converting from D3D11 to OpenGL, it is done using wglDXRegisterObjectNV + CopySubresourceRegion.

did it work?

Added a poc to
https://github.com/valbok/QtAVPlayer/blob/d3d11_gl/src/QtAVPlayer/qavhwdevice_d3d11.cpp#L140
this converts D3D11 textures with NV12 pixel format to 2 opengl textures, R8 and R8G8. But rendering I did not test. Maybe there is a bug.

it does not work for Qt6.3 when I tested, but potentially it should work in future when will be fixed in Qt.

i think i'm missing something here.. if i understand correctly, all the decoding happens on ffmpeg side, and qt is only responsible for presenting the content (using qml or qopenglwidget)

so by "it doesn't work for qt6.3" do you mean that auto texture = (ID3D11Texture2D *)(uintptr_t)av_frame->data[0]; returns an invalid handle or the actual ID3D11Texture2D can't be used in qt even with conversion?

Let me explain how "everything" works:

1. ffmpeg decodes a source using hardware acceleration and puts data to D3D11 or Metal or VDPAU or VAAPI or whatever textures.
2. Pointer to texture is returned in av_frame->data[x]
3. To get access to the data for the frame in this case you should use av_hwframe_transfer_data: https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideobuffer_gpu.cpp#L22 - it is done when you call QAVVideoFrame::map() --- it downloads data from gpu to cpu!

What should you do when you would like to render it without mapping in RHI?

1. F.e. using Metal textures: we convert QAVVideoFrame to QVideoFrame and set QVideoFrame::RhiTextureHandle - > defines that the frame contains raw texture handle.
   RHI knows that current platform is macOS and handleType() points to RhiTextureHandle so need to try to reuse this texture handle instead of calling QVideoFrame::map().

2. For Windows, D3D11 render is used by default. So if you use QVideoFrame::RhiTextureHandle , Qt is expecting to have pointer to ID3D11Texture2D.
   2.1 So if ffmpeg creates this texture for us, we can try to return it - > it does not work since pixel format is NV12 which expects to have 2 textures - color and data.
   2.2 Creating 2 instances of ID3D11Texture2D textures and copying data to it from original -> did not work for me yet -> Qt should reuse these textures and avoid mapping but crashes. Need to dive deeper, maybe doing something wrong.

For widgets -> you would need to create renders yourself to reuse D3D11 textures or convert to openGL as described above.

Sorry for long story.

from qtavplayer.

This is the output i get when playing a video (the video works and i do get vframes and audio):

h264 : supported hardware device contexts:
    dxva2
    d3d11va
    cuda
Creating hardware device context: d3d11va
Using hardware device context: d3d11va
Available pixel formats:
   yuv420p : AVPixelFormat( 0 )
   cuda : AVPixelFormat( 119 )
   dxva2_vld : AVPixelFormat( 53 )
   d3d11va_vld : AVPixelFormat( 118 )
   d3d11 : AVPixelFormat( 174 )
Using hardware decoding in d3d11

from qtavplayer.

Hi, thanks for the issue. Is it about Qt5?
It uses d3d11 but Qt does not support to render it copy-free, you are trying to use OpenGL here, so you would need to transfer d3d11 texture to OpenGL textures. Which is not currently implemented in QtAVPlayer. But maybe nice to have.
Currently QtAVPlayer tries to use Format_NV12 when converting to QVideoFrame which will download data from d3d11 textures. It is done like this because there was no way to render d3d11 textures anyway.

from qtavplayer.

Hello,
Nope i'm using qt 6.3

At the moment i'm trying to get the vframe data using map() method, the reason i'm using opengl is because software render with QPainter is super slow and makes the entire window lag.

Do you have any suggestions for my usecase?

from qtavplayer.

It is done like this because there was no way to render d3d11 textures anyway.

It seems like i reached a deadend :(

from qtavplayer.

Hello, Nope i'm using qt 6.3

At the moment i'm trying to get the vframe data using map() method, the reason i'm using opengl is because software render with QPainter is super slow and makes the entire window lag.

Do you have any suggestions for my usecase?

For Qt6 it might be possible to use RHI and d3d11 with QVideoFrame, when I checked last time in 6.3.0 it did not work. Need to dive into again.
toImage() is not necessary, try workaround QVideoFrame videoFrame = frame.convertTo(AV_PIX_FMT_RGB32); which like in example: widget_video , also need to check if it uploads data to OpenGL textures while using OpenGL renderer.

from qtavplayer.

toImage() is not necessary, try workaround QVideoFrame videoFrame = frame.convertTo(AV_PIX_FMT_RGB32); which like in example: widget_video , also need to check if it uploads data to OpenGL textures while using OpenGL renderer.

Alright, i'll try this and tell you.

small note: i'm using QAVVideoFrame directly because QVideoFrame crashes randomly when i either map() or unmap (something related to "destorying a locked QMutex").

from qtavplayer.

toImage() is not necessary, try workaround QVideoFrame videoFrame = frame.convertTo(AV_PIX_FMT_RGB32); which like in example: widget_video , also need to check if it uploads data to OpenGL textures while using OpenGL renderer.

Alright, i'll try this and tell you.

small note: i'm using QAVVideoFrame directly because QVideoFrame crashes randomly when i either map() or unmap (something related to "destorying a locked QMutex").

crashes should be fixed, do you use not the latest version?

from qtavplayer.

I'm using the master branch version (latest commit jul 24)

from qtavplayer.

would you please catch the crash and report it? thanks

from qtavplayer.

Sure, i will if i faced it again.

Altho it only happens when using either QVideoFrame::un/map() in QWidget/QOpenGLWidget::paintEvent()
It doesn't happen when using QAVVideoFrame, so i doubt that there is a bug in this library

from qtavplayer.

Altho it only happens when using either QVideoFrame::un/map() in QWidget/QOpenGLWidget::paintEvent()
It doesn't happen when using QAVVideoFrame, so i doubt that there is a bug in this library

It seems i was mistaken, a crash happens when using QAVVideoFrame::map() as well.

So i think it might be the same reason for crashes when using QVideoFrame :p

from qtavplayer.

For anyone looking for an example, scroll up

from qtavplayer.

Had a hope that if d3d11 renderer is used by default, it waits for ID3D11Texture2D texture,
auto texture = (ID3D11Texture2D *)(uintptr_t)av_frame->data[0];
returning this does not work. Because NV12 pixel format but texture is one. Qt will try to get textures per plane. Need to split it to 2 textures for R8 and R8G8? Does not work, crashes in QRHI.

from qtavplayer.

@valbok most of the videos i tried (mostly mp4@h264) use d3d11 pixfmt, so in theory auto texture = (ID3D11Texture2D *)(uintptr_t)av_frame->data[0]; should work, because it does use d3d11 under the hood

i think the way qt handles things is not important, since in qopenglwidget we are working with opengl directly
Something like this could fix our texture conversion problem but it requires minimum opengl 4.5, another option is to use google's ANGLE to translate textures, altho i wonder if it can actually do that

also i assume that qrhi is not ready for public use yet, at least in qtwidgets

from qtavplayer.

I already made a converting from D3D11 to OpenGL, it is done using wglDXRegisterObjectNV + CopySubresourceRegion.

i tried (mostly mp4@h264) use d3d11 pixfmt, so in theory auto texture = (ID3D11Texture2D *)(uintptr_t)av_frame->data[0]; should work,

it does not work for Qt6.3 when I tested, but potentially it should work in future when will be fixed in Qt.

from qtavplayer.

hello, sorry for late reply

I already made a converting from D3D11 to OpenGL, it is done using wglDXRegisterObjectNV + CopySubresourceRegion.

did it work?

it does not work for Qt6.3 when I tested, but potentially it should work in future when will be fixed in Qt.

i think i'm missing something here..
if i understand correctly, all the decoding happens on ffmpeg side, and qt is only responsible for presenting the content (using qml or qopenglwidget)

so by "it doesn't work for qt6.3" do you mean that auto texture = (ID3D11Texture2D *)(uintptr_t)av_frame->data[0]; returns an invalid handle or the actual ID3D11Texture2D can't be used in qt even with conversion?

from qtavplayer.

Sorry for long story.

NO!
Thanks a lot for the detailed explanation, now i understand how it works

2.1 So if ffmpeg creates this texture for us, we can try to return it - > it does not work since pixel format is NV12 which expects to have 2 textures - color and data.
2.2 Creating 2 instances of ID3D11Texture2D textures and copying data to it from original -> did not work for me yet -> Qt should reuse these textures and avoid mapping but crashes. Need to dive deeper, maybe doing something wrong.

I'll try to get this to work

Thanks again :)

from qtavplayer.

#371 Added native D3D11 Textures and RHI from 6.4.0

from qtavplayer.