On iOS devices, rendering of videotoolbox decoded frames stops after 30000 frames. about qtavplayer HOT 24 OPEN

PS. XCode Instruments does not show leaks, nothing.

from qtavplayer.

PS. XCode Instruments does not show leaks, nothing.

Interesting can we trust this? Btw, are you saying it might happening due to mem leak somewhere?

from qtavplayer.

PS. XCode Instruments does not show leaks, nothing.

Interesting can we trust this?

Yes, I think so, I have good experience with xcode Instruments, it is even more helpful than valgrind. What I cannot see is if memory is allocated, still accessible, but should have been released because it's not needed anymore. Meaning memory which is not leaked but just unused waste. I don't see that here.

Also I did let the code run on MacOS over night, more than 8 hours, and it's still running and memory usage is stable (according to Activity Monitor in MacOS). If there would be any significant memory leak it should have shown.
The code on MacOS is 100% the same, only difference is the targeting of MacOS instead of iOS in the .pro file.

Btw, are you saying it might happening due to mem leak somewhere?

Yes, but not RAM, not memory, but resources, something like file descriptors, file handles (everyting on unix/linux is a file). As the tests confirm every time, the freeze happens after almost exactly 30000 frames are sent to display. The correct value seems to be more 29980 or a little less.

It happens too when I pause playing in between, multiple pauses, and also when I switch the input source (which means resetting the Codecs in QtAVPlayer).

Whatever I do on the frame producing side seems not to matter. When a total of almost 30000 frames, as produced by videotoolbox, without frame format conversion, is sent to display with videoSink->setVideoFrame(), then it freezes the UI.

But the freeze is not 100%, there are very short updates, just about one frame, and some UI elements, every minute or so - I guess because some resources got available again, 2 or 3 file descriptors or similar. Such an update from time to time is unlikely if something crashed or got locked.

And while this is happening in the display thread, the threads handling download and processing of the data run and run without problems whatsoever.

from qtavplayer.

Unless there are better ideas, I will try to confirm if this is happening too when using standard QtMM mediaplayer with Qt 6.5.2, on iOS devices, or not.

• If it does happen -> Open bug with Qt.
• If it does not happen -> What is different, what are they doing differently.

(This will be time consuming, I don't like to go that route, but is here a choice?)

from qtavplayer.

https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavhwdevice_videotoolbox.mm might be possible that some resources there should be cleaned up or reused. F.e. textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); also might be device m_hw->device ?

This is very suspicious CVPixelBufferRetain(m_hw->pbuf);
because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

from qtavplayer.

https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavhwdevice_videotoolbox.mm might be possible that some resources there should be cleaned up or reused. F.e. textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); also might be device m_hw->device ?

This is very suspicious CVPixelBufferRetain(m_hw->pbuf); because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

Yes, exactly, there might be the problem hidden somewhere. I have not much experience with Objective C, but I'll think through the code there.

from qtavplayer.

This is very suspicious CVPixelBufferRetain(m_hw->pbuf); because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

As a test, I commented out the line with CVPixelBufferRetain(m_hw->pbuf); It seemed to be not useful anyway, because m_hw->pbuf doesn't go out of scope.

// TEST: is this needed???        CVPixelBufferRetain(m_hw->pbuf);

However when testing on Mac, it crashes directly after showing the first frame:

Thread 7 Crashed:: QSGRenderThread
0   CoreFoundation                	       0x19643a8c4 _CFRelease + 216
1   libavcodec.59.37.100.dylib    	       0x10229b154 0x101e04000 + 4813140
2   Player                         	       0x100133704 VideoBuffer_MTL::~VideoBuffer_MTL() + 68
3   Player                         	       0x100125f90 QAVVideoFramePrivate::~QAVVideoFramePrivate() + 52
4   Player                         	       0x1001249f0 QAVStreamFrame::~QAVStreamFrame() + 40
5   Player                        	       0x100126080 PlanarVideoBuffer::~PlanarVideoBuffer() + 52
6   QtMultimedia                  	       0x101585f50 QVideoFrame::operator=(QVideoFrame const&) + 92
7   QtMultimediaQuick             	       0x1010a49f0 0x101098000 + 51696
8   QtMultimediaQuick             	       0x1010a4b8c 0x101098000 + 52108
9   QtQuick                       	       0x10393422c QSGBatchRenderer::Renderer::updateMaterialDynamicData(QSGBatchRenderer::ShaderManagerShader*, QSGMaterialShader::RenderState&, QSGMaterial*, QSGBatchRenderer::Batch const*, QSGBatchRenderer::Element*, int, int) + 168
10  QtQuick                       	       0x1039356ec QSGBatchRenderer::Renderer::prepareRenderMergedBatch(QSGBatchRenderer::Batch*, QSGBatchRenderer::Renderer::PreparedRenderBatch*) + 720
11  QtQuick                       	       0x103937c30 QSGBatchRenderer::Renderer::prepareRenderPass(QSGBatchRenderer::Renderer::RenderPassContext*) + 1692
12  QtQuick                       	       0x103937514 QSGBatchRenderer::Renderer::render() + 36
13  QtQuick                       	       0x10394cf28 QSGRenderer::renderScene() + 344
14  QtQuick                       	       0x103902a34 QQuickWindowPrivate::renderSceneGraph() + 780
15  QtQuick                       	       0x103a8eab4 0x103818000 + 2583220
16  QtQuick                       	       0x103a8f980 0x103818000 + 2587008
17  QtCore                        	       0x103f16f84 0x103d3c000 + 1945476
18  libsystem_pthread.dylib       	       0x196287fa8 _pthread_start + 148
19  libsystem_pthread.dylib       	       0x196282da0 thread_start + 8

So the CVPixelBufferRelease without prior CVPixelBufferRetain crashes it.

So now I test (again first on the Mac) with both commented out.
And it doesn't crash. And memory usage doesn't shoot up.
I'll let it run for a while and watch.

from qtavplayer.

pbuf seems ok than, but newTextureWithDescriptor interesting does it require to release this texture, or it creates everytime new.

from qtavplayer.

if there is a change in memory usage, then it seems to be slightly less, but not more.

Other than that, unchanged, it runs on Mac forever, but on iPad the UI freeze still happens. This time after exactly 29800 frames sent to display. The CVPixelBufferRelease() / CVPixelBufferRetain() does not cause it.

from qtavplayer.

pbuf seems ok than, but newTextureWithDescriptor interesting does it require to release this texture, or it creates everytime new.

            textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);

That line above?
I don't understand it yet, but yes, this looks like being the next candidate to examine.

from qtavplayer.

To check, if it might stop the freezing if that textures line would not run, I commented them out:

// TEST:  MTLTextureDescriptor *desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:f width:w height:h mipmapped:NO];

// TEST:  textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);

I expected that the video might play, just no picture showing, but to my surprise. the images show... video plays.

The textures returned are as set earlier there:

        QList<QVariant> textures = { 0, 0 };

nevertheless, the pictures are displayed. This code might not be needed, at least not in the test case, Qt 6.5.2 on real iPad device.

from qtavplayer.

It passed the 32000 frames without freezing !!!!!!

If textures are not needed, possibly nothing in that function is really needed, so new test with the complete function changed to this:

    QVariant handle(QRhi */*rhi*/) const override
    {
        QList<QVariant> textures = { 0, 0 };
        return textures;
    }

Nevertheless the video still plays fine and smooth.
I'll let it run now longer to check if there might be still freezing, just later.

from qtavplayer.

QList textures = { 0, 0 };

Exactly means that no textures are used and no HW accel for rendering. Means textures are leaking here.

from qtavplayer.

QList textures = { 0, 0 };

Exactly means that no textures are used and no HW accel for rendering.

You mean that the QtMM code handling the display (which called handle()) sees that there are no textures returned, and switches to a different way of rendering? Very probable, and if needed, verifiable in QtMM source.

Rendering seems to be very efficient though, may be because the image to render is already in GPU memory?

In any case, it plays perfectly smooth on the devices I tested so far, iPhone XS Max, iPhone 11 PM, 4th gen iPad 11 Pro. On a very old iPad 12 pro it stutters, but that might have other reasons, I still have to dig in there.

If this performance is confirmed over all iOS devices then it's good enough for use.

Means textures are leaking here.

Yes, but that's then in QtMM code, right?
Or, as a workaround, can we keep a handle on these textures and release their memory later, after they had been displayed?

from qtavplayer.

You mean that the QtMM code handling the display (which called handle()) sees that there are no textures returned, and switches to a different way of rendering? Very probable, and if needed, verifiable in QtMM source.

Yes, this is how it is implemented, it checks for texture first, if nothing, gets back to map() which downloads data from GPU if any.

from qtavplayer.

Means textures are leaking here.

Yes, but that's then in QtMM code, right?
Or, as a workaround, can we keep a handle on these textures and release their memory later, after they had been displayed?

No, it is somewhere here, we are responsible to free resources, so I will try to look a bit later, might be possible need to release textures in dtor

from qtavplayer.

does it mean that MacOS does not reproduce the issue? regardless how long it works?

from qtavplayer.

does it mean that MacOS does not reproduce the issue? regardless how long it works?

Yes, the issue does

• not show on MacOS, and
• not on the iOSSimulator (but that cannot use videotoolbox).

from qtavplayer.

could you check if you return empty textures
QList textures = { 0, 0 };

but keep calling textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); ?

trying to understand where it should be cleared.

from qtavplayer.

maybe it needs double CVPixelBufferRelease(m_hw->pbuf); ?

from qtavplayer.

or there is an issue inside QtMM, can you try just QtMM?

from qtavplayer.

could you check if you return empty textures QList textures = { 0, 0 };

but keep calling textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); ?

trying to understand where it should be cleared.

With this the freeze still happens at the same frame count.

    QVariant handle(QRhi */*rhi*/) const override
    {
        QList<QVariant> textures = { 0, 0 };
        QList<QVariant> textures2 = { 0, 0 };

        CVPixelBufferRelease(m_hw->pbuf);
        m_hw->pbuf = (CVPixelBufferRef)frame().frame()->data[3];
        CVPixelBufferRetain(m_hw->pbuf);

        if (!m_hw->pbuf)
            return textures;

        if (CVPixelBufferGetDataSize(m_hw->pbuf) <= 0)
            return textures;

        auto format = CVPixelBufferGetPixelFormatType(m_hw->pbuf);
        if (format != '420v') {
            qWarning() << "420v is supported only";
            return textures;
        }

        if (!m_hw->device)
            m_hw->device = MTLCreateSystemDefaultDevice();

        IOSurfaceRef surface = CVPixelBufferGetIOSurface(m_hw->pbuf);
        int planes = CVPixelBufferGetPlaneCount(m_hw->pbuf);
        for (int i = 0; i < planes; ++i) {
            int w = IOSurfaceGetWidthOfPlane(surface, i);
            int h = IOSurfaceGetHeightOfPlane(surface, i) ;
            MTLPixelFormat f = i ?  MTLPixelFormatRG8Unorm : MTLPixelFormatR8Unorm;
            MTLTextureDescriptor *desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:f width:w height:h mipmapped:NO];

            textures2[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);
        }

        return textures;
    }

I only added textures2, so that textures can get returned empty and textures2 will get filled.

I would have expected that textures would/should get released later, in the code which calls handle(). As far as I can see, here we can only return them, and don't see them again.

from qtavplayer.

maybe it needs double CVPixelBufferRelease(m_hw->pbuf); ?

where would you insert that?

from qtavplayer.

or there is an issue inside QtMM, can you try just QtMM?

I never had such freezing when using QtMM / QMediaPlayer alone.

I have my old stuff running with Qt 5.15.x and also Qt 6.4.3 QtMM, using QMediaplayer in C++ linked to QML Videooutput. The QMediaPlayer part is now replaced with QtAVPlayer, otherwise almost identical, thus comparable.

This freezing issue did not show up, never. It was extensively tested on real iOS devices also with Qt 6.4.3. What I experienced there was on some devices almost flawless playing, but on many devices often stop and go. QtMM is calling the IOS internal Player, and somewhere there it got hiccups, often, too often. Also when the input has timecodes which are not always perfect, such as when you load streams over the internet and some parts get lost, or the stream is bad quality, or input resetted, then QtMM and the iOS internal Player always stumble.

This is all fixed now with using QtAVPlayer/ffmpeg, it plays for hours without issues. What I hoped for when I stumbled over QtAVPlayer, it got fulfilled, finally a stable Player for Android and iOS.

(Disclaimer: I have modified the PTS syncing a little to adapt to input from constant streams with sometimes messed up PTS values. Such an adaption was not possible with QtMM, but with QtAVPlayer all is possible. Thanks for that!).

from qtavplayer.