Idea: switch based on content-match of screen rect about sceneswitcher HOT 11 CLOSED

Let me describe a similar but simpler function and what it would be used for -- simply switching based on the colour of a pixel. The application is streaming slide presentations and screen demos where you are mixing in the slides with webcam video or other sources. On some slides you want the webcam video to appear in the corner. Other times you want it to fill the screen and not show the slide. Other times you don't want to see it at all. You may want to change its size or grow it etc. Sometimes greenscreen me into the slides. Other times, put a background behind me as it doesn't blend well. Etc.

This tool would allow you to put "clues" in the slides, single pixel clues, as to what scene is right for that slide. Thus the user would just play the deck and their video would be mixed correctly. Playing a video -- get rid of my face. Time to cut to just me, put in a slide with the right colour. Got a slide where I've left room on the right side for my face -- put in a version that fills that region etc. Even switch camera angles based on what slide I am showing.

Unless you can think of a better and easier way to signal a scene change. Since slide programs are different, it's better not to use one tied to powerpoint (though its speaker notes could be used for the clue pixels.)

Of course, if you wanted to get fancy you could attempt to OCR a rectangle and read the scene name from it, which would be put there in very small type. Or match the specific image rectangles as suggested above. One could either have automatic naming of scenes like SS1F1F1E for particular pixel names (which is not very easy to read) or have a UI or a text file to pair pixel values with scene names. I suspect most people would just use a pixel in the corner of their window, which may be cropped by the capture device so hopefully you access the real frame buffer.

from sceneswitcher.

Just for future reference.
HoneyHazard is working on a separate plugin for this.
https://github.com/HoneyHazard/PixelMatchSwitcher

from sceneswitcher.

FYI - I was the original requester, and I specced out and funded HoneyHazard's development work, because I think this would be a useful addition to OBS. Pixel Match Switcher is already quite usable. We want to get it out to the community, so hopefully you can help support getting it used and tested and eventually merged into OBS!

from sceneswitcher.

@bradtem - Assuming the pixel-locations in the slides are stable, you can use PixelMatchSwitcher to do what you want now. I encourage you to give it a try. You could either put small color code in the corner to match on, or you could match on specific "video location boxes" in the slides.

https://github.com/HoneyHazard/PixelMatchSwitcher/releases/tag/0.14-alpha

As a future followup, it might not be too hard to make a related but different plugin that looks for a specific chroma-colored rectangle anywhere in a source, and relocates a video source into it... though I suspect it would be a different plugin and development effort... After we get OBS wrapped up I'll talk to HoneyHazard about it.

from sceneswitcher.

Good to hear about. I recently saw somebody who built this a different way -- they used the remote control API for OBS and would read the speaker notes in powerpoint to look for scene name commands, and send them to OBS. That's very powerpoint specific, but the interface for the user is of course nicer than putting in the coloured dot. But I don't use powerpoint anyway.

I look forward to this getting out of alpha. My normal build environment is linux but I run OBS on windows. Looks like right now it replaces your whole OBS and is not yet a loadable extension?

from sceneswitcher.

PixelMatchSwitcher is a plugin extension, but it currently requires a custom build of OBS.. because in order to implement the plugin, we needed to add things to the GPU effect language built into OBS. It's basically support for atomic GPU counters, so our effect shader can count matching pixels and communicate that back to the CPU code.

We hope it won't take long for these patches to be accepted into OBS proper once we publicly announce PixelMatchSwitcher on the forums and initiate a pull request.. once these changes are accepted, PixelMatchSwitcher itself can be "just a plugin".

We're just about ready to do a public forum post. We're tentatively planning on this coming Tuesday.

HoneyHazard has been working on PixelMatchSwitcher for a few months now, and it's been a pretty good chunk of work.

For the curious, you can take a look at our OBS atomic counters fork here.... (when we are ready we will make a proper pull request to OBS)

• https://github.com/HoneyHazard/obs-studio-atomic-effects/tree/atomic-effects
• obsproject/obs-studio@master...HoneyHazard:atomic-effects

from sceneswitcher.

I thought one way to do it would be to supply a filter. Filters get access to the frame buffer and normally modify it, but don't need to. A filter could just examine it, and send messages to a scene switcher so it would be portable, no GPU code.

from sceneswitcher.

I thought one way to do it would be to supply a filter. Filters get access to the frame buffer and normally modify it, but don't need to. A filter could just examine it, and send messages to a scene switcher so it would be portable, no GPU code.

We indeed use a filter, but we use atomic counters in the effect/shader for a greater performance than frame readbacks will ever give us. (frame readbacks are used but only for things like extracting region images from the filter's output, as a source of match image or match mask) @jeske wanted a plugin that is fast, and nothing would beat shaders.

One could mention that OBS already does a lot of specialized GPU things in reaction to effects. We made effects do a little more and we now do more GPU things to support that. Just like before, you write the same effect code that is agnostic to whether D3D11 or OpenGL is used.

I look forward to this getting out of alpha. My normal build environment is linux but I run OBS on windows. Looks like right now it replaces your whole OBS and is not yet a loadable extension?

Like @jeske said, it is a loadable extention but right now you need a special fork of OBS to support the plugin.

I fixed a few bugs yesterday and was doing successful builds on Debian 10 yesterday. There are Debian instructions for building the whole tree (atomic effects fork of OBS + plugin). This involved the fork made specifically for this purpose, and it is the easiest way to get things building and installing. You can also build the plugin outside of the OBS source tree, but you still need the atomic-effects fork to support atomic counters.

from sceneswitcher.

We actually experimented with readback and CPU matching, but this method didn't match a single match test region fast enough to switch in the same frame.. while optimization and multithreading might have been able to overcome that, it would not have allowed many simultanes match tests to happen every frame, and all that CPU use would severely affect laptop battery life.

GPU matching is very efficient...can have many match tests tested every frame, and still match and switch in the same frame. And laptop battery usage is minimal.

Matching every frame and switching in the same frame is important for our primary use case.. which is allowing game streamers to trigger mask coverups the prevent leaking sensitive data into their game stream for even a singlrle frame.

from sceneswitcher.

I now added very basic functionality like this to the plug-in with e4bb034.
It works by continuously making screenshots of the selected source and comparing it to the target image.

@HoneyHazard I hope you don't mind me linking to your plug-in for a much better implementation of this functionality.

from sceneswitcher.

@HoneyHazard I hope you don't mind me linking to your plug-in for a much better implementation of this functionality.

Not at all. Open source all the way! I have learned a lot from your plugin, and will be happy to know if you've found some use for our work.

from sceneswitcher.