Stream stops if the tab is minimized about video-stream-merger HOT 20 CLOSED

They ARE supposed to be throttled, but you can violate the budgeting policy by playing audio - which is what we do in this library. It's a hack, but it's the only way to get solid framerates in background tabs.

from video-stream-merger.

Maybe, haven't seen that before.

from video-stream-merger.

Should be fixed in v3.3.1. I expect this issue with budget throttling to come up again, but it works for now.

from video-stream-merger.

index.html

<button onclick="startCapture()">Start</button>
<button id="stop">stop</button>
<select id="param">
  <option value="0">0</option>
  <option value="0.1">0.1</option>
  <option value="0.2">0.2</option>
  <option value="0.3">0.3</option>
  <option value="0.4">0.4</option>
  <option value="0.5" selected="selected">0.5</option>
  <option value="0.6">0.6</option>
  <option value="0.7">0.7</option>
  <option value="0.8">0.8</option>
  <option value="0.9">0.9</option>
  <option value="1.0">1</option>
</select>
<h2 id="msg">wait</h2>
<video
  playsinline
  id="video"
  autoplay
  controls
  style="object-fit: contain; width: 40%;"
></video>

<script src="./video-merge.js"></script>
<script src="https://cdn.WebRTC-Experiment.com/RecordRTC.js"></script>

<script>
  let videoElem = document.getElementById("video");

  async function startCapture() {
    var CANVAS_MULTIPLIER = 5;
    try {
      var screen = await navigator.mediaDevices.getDisplayMedia({
        video: true
      });
      console.log(screen);

      var cam = await navigator.mediaDevices.getUserMedia({
        audio: true,
        video: true
      });

      const select = document.getElementById("param");
      const param = select.options[select.selectedIndex].value;

      console.log(2 - param);
      var merger = new VideoStreamMerger({
        width: window.screen.width / (2 - param),
        height: window.screen.height / (2 - param)
      });

      setTimeout(function() {
        console.log("trigged");
        merger.addStream(screen, {
          x: 0, // position of the topleft corner
          y: 0,
          width: merger.width,
          height: merger.height,
          mute: true // we don't want sound from the screen (if there is any)
        });
        console.log("trigged2");

        merger.addStream(cam, {
          x: 0,
          y: merger.height - 0.2 * merger.width,
          width: 0.2 * merger.width,
          height: 0.2 * merger.width,
          mute: false
          // draw: function (ctx, frame, done) {
          //   x = 0; y = merger.height - 0.2 * merger.width; width = 0.2 * merger.width; height = 0.2 * merger.width;
          //   // decide where you want your circular image, and what size

          //   ctx.save(); // save canvas context
          //   ctx.beginPath();
          //   ctx.arc(x + width/2, y + height/2, width/2, 0, Math.PI*2, true);   // create an circle centered around frame
          //   ctx.closePath();
          //   ctx.clip(); // clip the context to the circle

          //   ctx.drawImage(frame, x, y, width, height); // draw the image in the clipped context
          //   ctx.restore(); // restore canvas context so you don't clip every other stream
          //   done() // <- you must call this so the merger can continue
          // }
        });

        merger.start();

        // We now have a merged MediaStream!
        // videoElem.srcObject = cam;
        // videoElem.srcObject = screen;

        videoElem.srcObject = merger.result;

        document.getElementById("msg").innerHTML = "Start";
        // replace merger.result with cam or screen
        var recorder = new RecordRTCPromisesHandler(merger.result, {
          type: "video"
        });

        recorder.startRecording();
        var stop = document.getElementById("stop");
        stop.onclick = async () => {
          console.log("stopping recording");
          await recorder.stopRecording();
          let blob = await recorder.getBlob();
          invokeSaveAsDialog(blob);
        };
      }, 1000);
    } catch (err) {
      console.error("Error: " + err);
    }
  }
</script>

and place the latest version of this library in the same folder as video-merge.js
to change the sources being recorded, check the line numbers 89-91 (uncomment as required) and line number 95

from video-stream-merger.

It has to do with the requestAnimationFrame call when merging a stream, the canvas is throttled when the tab is no longer focused. There currently isn't a way around this, other than maybe https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas. I would look into changing video-stream-merger's code to use that to fix this.

from video-stream-merger.

I see. It works fine when the screen is out of focus, that is when I drag another window over the browser. This freezing of stream only happens when the browser is explicitly minimized. Do you think it is consistent with throttling?

from video-stream-merger.

I tested this, it looks like the hack of playing audio to prevent throttling has been fixed in the latest Chrome. Tabbing out also causes the issue.

I'll have to look at what Spotify is doing now to avoid throttling.

from video-stream-merger.

3.2.1 use setTimeout when the page is hidden. It's not as resource efficient, but the stream will continue merging.

from video-stream-merger.

@t-mullen I am still experiencing this issue. Having a look at the fix you used, my problem seems fixed when I rewrite _requestAnimationFrame() to this:

// Wrapper around requestAnimationFrame and setTimeout to avoid background throttling
VideoStreamMerger.prototype._requestAnimationFrame = function (callback) {
    setTimeout(callback, 0);
}

Weird how I don't need an explicit requestAnimationFrame() but it just works. I don't know if the latest version of Chrome somehow got more strict, essentially dismantling your fix or if my specific scenario is different from what was originally reported. Could you have a look if what I'm saying makes sense?

from video-stream-merger.

Chrome might be throttling the minimized tab so much that document.hidden isn't even being run once to switch to the fallback. I'll change this so that we're checking minimization in a setTimeout timer and never relying on a requestAnimationFrame timer returning.

Just using setTimeout works, but requestAnimationFrame has some nice optimizations for canvas that let us get a more stable framerate.

from video-stream-merger.

Awesome, sounds like a great fix. I'm relying on just the setTimeout through patch-package in my current application but will move to your solution once implemented. Thanks for this!

from video-stream-merger.

@Timebutt How are you using setTimeout(callback, 0) without completely blocking the main thread? I've tried this on a local install and the browser seems to crawl to a halt when using this since the callback is being called very quickly repeatedly.

from video-stream-merger.

@Tryptophan you are absolutely right. While it does work when you alt-tab, it fully loads the main thread. I'm on a powerful 6 core machine and didn't even notice it does, trying this again on a less beefy machine is a whole different story.

Thanks for pointing this out, looks like we'll need the solution @t-mullen is suggesting: using the setTimeout() to do the minimisation check and continue to use requestAnimationFrame(). Any word when you might have a chance to fix this?

from video-stream-merger.

Still working on a fix. The visibility API seems to lie about tab visibility when Chrome switches tabs...

from video-stream-merger.

It seems my audio hack no longer works. Spotify is still able to maintain intervals when it is playing... let's hope Chrome hasn't implemented a domain whitelist for timers.

from video-stream-merger.

Just so you're aware, I think setTimeout and setInterval are still throttled in background tabs: https://stackoverflow.com/questions/6032429/chrome-timeouts-interval-suspended-in-background-tabs

from video-stream-merger.

Thanks for this @t-mullen, I'll have a look at how you fixed it and verify if performance is back where it's supposed to be.

FYI: I also had issues with the Visibility API in another problem I was fixing, it has to do with the OS you are on among others it seems.

from video-stream-merger.

I am not sure that the solution has covered the case where OSX users totally minimize (- button) chrome application and the render stop working.

I have solved this long time ago, but I cant remember the actual thing I did. I used some kinds of Worker running in background to call the drawing function. And the drawing function keeps signaling the worker for settimeout. So the render is running smoothly according to the fps without interuption.

Basically, its a settimeout worker running in background. U can also use this worker for other features to trigger a function in the background.

from video-stream-merger.

This is issue still persist in Safari, seems to work fine in chrome

from video-stream-merger.

This is issue still persist in Safari, seems to work fine in chrome

The solution is to tell your users to stop using Safari, if you can convince them to do it that would also help many other websites as well; due the millions of other safari-bugs that Apple refuses to address, at this point Safari is the new IE6

from video-stream-merger.