Integrate with Web Audio about mediasession HOT 10 CLOSED

Also, in the media session algorithms (starting here in the spec) we would additionally handle 'active participating web audio objects' of a media session wherever we also currently handle existing 'active participating media elements'.

from mediasession.

@richtr I think it would be better to introduce some concept like "audio producer" and say that media elements and audio contexts both have an audio producer. Maybe eventually audio producer gets defined in terms of web audio, e.g. as an audio worker.

The net effect of this would be to isolate the interface points between media session and media elements/audio contexts down to just a single section in the spec, instead of spreading it throughout.

from mediasession.

Great idea @domenic!

So I will work on replacing 'participating media elements' with a generic like 'participating audio producers'.

The net effect of this would be to isolate the interface points between media session and media elements/audio contexts down to just a single section in the spec, instead of spreading it throughout.

I will then work on defining interaction of media sessions with audio producers in separate algorithms (e.g. 'pause an audio producer', 'unpause an audio producer', 'duck an audio producer', 'unduck an audio producer', etc).

Right now those algorithms will probably treat AudioContext and HTMLMediaElement objects separately (e.g. 'For each AudioContext-based audio producer...suspend() the audio context', 'For each HTMLMediaElement-based audio producer...pause() the media element', etc). We could then eventually replace those algorithms with something based on AudioWorker or whatever our unified 'audio producer' ends up being.

Does this sound good?

from mediasession.

That does sound pretty good!

For clarity I think it would make sense to try to isolate the AudioContext and HTMLMediaElement stuff to their own sections. So instead of defining "to duck an audio producer... if it's AudioContext-based, do X; if it's HTMLMediaElement-based, do Y", you'd say stuff like:

Audio producers are an abstract concept representing audio production blah blah. They have a number of operations:

• ...
• Ducking, which conceptually means lowering the volume to get out of the way
• ...

We currently define two audio producers, which implement these procedures in their own idiosyncratic ways. In the future we could maybe unify blah blah.

Then later you'd have a section for "The AudioContext audio producer" which says something like:

An AudioContext is a type of audio producer. It implements the audio producer operations as follows:

• To duck an AudioContext, do blah blah...

from mediasession.

Going back to the OP issue:

An important difference is that new AudioContext() creates an audio context which is already playing (silence), so it would be like connecting a playing media element to a session, which we haven't wanted to do.

It may make sense to disallow web developers from assigning a media session to an audio context unless its state attribute is 'suspended'. Or let web developers somehow create an AudioContext object that is born 'suspended'.

Something like the following could work:

// Web Audio

var ctx = new AudioContext(); // Go!! (with no media session)

ctx.session = session; // will throw an exception because ctx.state !== "suspended"

ctx.suspend().then(() => {
  ctx.session = session; // will now work because ctx.state === "suspended"

  // ... optionally, load data ready for our AudioContext to start

  return ctx.resume(); // Go!! (with a media session)
}).then(() => {
  console.log("ctx now belongs to an active media session");
});

The good thing here is that an AudioContext object will then only grab audio focus when .resume() is invoked allowing the web page to do any audio context setup it may need before it becomes ready to play and grabs audio focus away from other media resources e.g. pre-configure the media session controls they want to show, pre-buffer content from the network or otherwise setup source nodes, etc.

It will also allow us to make the way media sessions work on AudioContext and HTMLMediaElement identical. Here's how the same principle would be applied to HTMLMediaElement objects:

// HTMLMediaElement

var el = document.createElement('audio');
el.play(); // Go!! (with no media session)

el.addEventListener('playing', () => {
  el.session = session; // will throw an exception because el.paused === false

  el.pause();

  el.addEventListener('pause', () => {
    el.session = session; // will now work because el.paused === true
    el.play(); // Go!! (with a media session)
  }, false);
}, false);

tl;dr Attempting to assign a MediaSession to an AudioContext object when its active attribute is 'running' or 'closed' (i.e. not 'suspended') will throw an exception. Attempting to assign a MediaSession to an HTMLMediaElement object when its paused attribute is false (i.e. it is running) will throw an exception.

from mediasession.

I think the implicit activation makes more sense for HTMLMediaElement than AudioContext. It's a bit odd to have to create an audio context, suspend it, assign a media session, and then resume it. A model where an audio context has a single media session over its entire lifetime seems simpler. (Doesn't work as well a media element because it can play multiple resources during its lifetime, it's wrapping an underlying player that's created and destroyed, which is the object which ought to have a single session over its lifetime.)

This is related to a proposal about exposing MediaSession.activate() that we've been discussing in the office, and I'll create a separate issue for that.

from mediasession.

Let's try to keep Web Audio discussion in this issue, but in #50 I've proposed a change that I believe would make things slightly simpler for Web Audio.

from mediasession.

It seems that we agree that the AudioContext should have a session attribute. Could we reflect that in the specification? We are currently sending the signal that WebAudio is a second class citizen in this specification.

from mediasession.

I'm going to add something similar to #48 (comment) to the specification this week minus the throwing of exceptions if an AudioContext object is in the 'wrong' state (e.g. 'running').

The main difference from #48 (comment) then is that the web developer does not need to manually suspend() an AudioContext object before assigning a MediaSession object to it as that will be done internally according to the spec algorithms. i.e. we will not throw an exception if an AudioContext object is 'running' when we assign it to a new MediaSession object, instead we will suspend it.

When a new MediaSession object is attached as AudioContext object's session then it can be activated by calling resume() at any later point on the AudioContext object.

Here's an example:

// Web Audio

var ctx = new AudioContext(); // Go!! (with no/default media session)

// ctx.state === "running"

ctx.session = new MediaSession("content");

// ctx.state === "suspended"

ctx.resume(); // Go!! (with a media session & activate that media session)

// ctx.state === "running"

This matches the current specification logic for attaching a new MediaSession object to a currently playing HTMLMediaElement object via its session attribute (i.e. a HTMLMediaElement object is paused when its session is changed) and also reflects nascent implementation experience of that convention in Webkit: https://bugs.webkit.org/show_bug.cgi?id=147493.

from mediasession.

Added in e62c0d5.

from mediasession.