What are the keyboard events? I tested this yesterday, and on a Mac no events are delivered when pressing these buttons.

Also, on what object should those key events be dispatched? Having them on the document would require some page-provided framework to delegate if there are multiple components which can play audio. Android's MediaSession is documented as "In general an app only needs one session for all playback, though multiple sessions can be created to provide finer grain controls of media." Allowing the same in a Web API seems like a good starting point.

Finally, there's the "Double tap and other key sequences may follow platform-specific conventions" bit in the README.

from mediasession.

http://www.w3.org/TR/DOM-Level-3-Events-key/#keys-multimedia
http://www.w3.org/TR/DOM-Level-3-Events-code/#key-media

from mediasession.

Also, on what object should those key events be dispatched?

On the element that has focus. The bit we're missing is a way to lock specific keys (the media ones) to a particular element.

If an <audio> or <video> has focus, I think it's pretty reasonable to allow media keys to control playback. So, if we had a way to give an element "global media key focus", applying it to <audio> or <video> gives you a lot of behaviour by default.

However, the developer could add listeners, prevent the default & do their own thing. Allowing an arbitrary element to get a global focus for these events means it can be used for playback driven by SVG, Web Audio, Canvas, Web Animation etc etc.

Finally, there's the "Double tap and other key sequences may follow platform-specific conventions" bit in the README.

I think platform defaults are fine but developers should be able to prevent default and do their own thing. Extensible web & all that.

from mediasession.

Thanks @annevk! I find the string "MediaPlayPause" in Blink so presumably there is a platform where it works, probably Windows.

If there are sites (YouTube?) using these key events then we have a bit of a problem, in particular if it's the MediaTrackNext/MediaTrackPrevious keys where we couldn't just implement default behavior to do the right thing and stop firing the events.

Android has events delivered via RemoteControlClient or MediaSession:
https://developer.android.com/reference/android/media/RemoteControlClient.html
https://developer.android.com/reference/android/media/session/MediaSession.html
https://developer.android.com/reference/android/view/KeyEvent.html

Note the inclusion of both KEYCODE_MEDIA_FAST_FORWARD and KEYCODE_MEDIA_NEXT, as well as e.g. stuff related to rating.

iOS has MPRemoteCommandCenter:
https://developer.apple.com/library/prerelease/ios/documentation/MediaPlayer/Reference/MPRemoteCommandCenter_Ref/index.html

Most of the commands fall outside the existing key events. That doesn't mean we can't add key events, of course.

from mediasession.

Hi @jakearchibald!

On the element that has focus.

This sounds interesting, can you submit the proposal for Element-based audio focus so that we can take a look? (You didn't mention audio focus, but I assume there's some connection.)

If an <audio> or <video> has focus, I think it's pretty reasonable to allow media keys to control playback.

Absolutely, not making this work would amount to a complete failure. It's even the first use case mentioned in the README: "While watching a video or listening to audio in the background, pause and resume playback using the play/pause key."

There are many ways to achieve this modest goal of course. MediaRemoteControl has a RemoteKeyEvent and MediaSession is half-baked in this regard, but implies simple events like 'previous' and 'next' fired on the MediaSession object.

from mediasession.

Basic usage

Even if the device is incapable or unwilling to grant media focus, you want the keys to work when the page is focused:

var audioElement = document.querySelector('audio.music');

document.addEventListener('keyup', event => {
  switch (event.key) {
    case "MediaTrackNext":
      event.preventDefault(); // etc etc
      break;
    case "MediaTrackPrevious":
      event.preventDefault(); // etc etc
      break;
    case "MediaPlayPause":
      event.preventDefault(); // etc etc
      break;
  }
});

To request media focus:

audioElement.requestMediaFocus();

If the focus is granted, the media keys will be dispatched from audioElement even if the browser doesn't have focus & the tab isn't visible. These events will bubble up to the document and work as intended, meaning the developer doesn't have to repeat event handlers.

requestMediaFocus is on Element, meaning it can be used for media experiences created with lower-level components.

Question: Should requestMediaFocus require an interaction event like requestFullscreen? A play button click or MediaPlayPause key event would be ideal for this.

Controlling media focus

requestMediaFocus resolves to an instance of MediaFocusController for the purposes of releasing focus and setting details.

element.requestMediaFocus().then(function(mediaFocusController) {
  // To set metadata
  mediaFocusController.setDetails({
    poster: mediaImage,
    title: "Song title",
    duration: durationInMs
  });

  // Listen for loss of focus:
  mediaFocusController.addEventListener('released', func);

  // To release focus:
  mediaFocusController.release();
}, function() {
  // Focus denied
});

The user agent may infer details from the element until setDetails is called by the developer. For an audio element this could mean getting titles and imagery from ID3v2 etc and updating them as src changes (playback position could continually update).

Key defaults & automatic media focus

Pressing MediaPlayPause while an audio/video element has focus (normal element focus) should play/pause the media as a default, overridable using key events & preventDefault(). Since media focus uses key events, media keys inherit sensible defaults there too.

A UA may choose to assign media focus to an element automatically. Eg, if a mobile user plays video/audio on a page, then locks the screen, the UA could media-focus the playing element. The default key events come into play, as does the default detail detection.

This plays well with .requestMediaFocus(), which would return an existing (but internal) instance of mediaFocusController if the element already has media focus.

from mediasession.

(btw MediaSession is a way better name for MediaFocusController - I just chose something different to make it clear I wasn't reusing the one in the repo)

from mediasession.

Thanks Jake, that's interesting, and it points to a use case that isn't listed in the README, something like "If nothing on the system has media focus, such that pressing the hardware media keys would have no effect, allow the foreground app/tab to respond to those keys, e.g. to begin playback." Does that sound right?

In the MediaSession proposal this isn't possible, because media key events are only delivered when you have audio focus. If there were a way to request audio focus with the lowest possible priority, such that it couldn't interrupt anything at all, that might work. However, I don't think there's any way to do that on Android.

One could also make audio focus and key events entirely orthogonal concepts, but I think that would lead to some pretty strange behavior as seen from a user. Sorting out #9 would go a long way towards figuring out the basic design.

(There's also the question of how you get a MediaSession or MediaFocusController object, either you can simply create them or you get them from a promise. The answer really depends on how the cycle of gaining-losing-and-gaining focus is to be represented. MediaSession starts out in the same state as if it had lost focus, while I guess one would have to request a new MediaFocusController.)

from mediasession.

@foolip

Thanks Jake, that's interesting, and it points to a use case that isn't listed in the README, something like "If nothing on the system has media focus, such that pressing the hardware media keys would have no effect, allow the foreground app/tab to respond to those keys, e.g. to begin playback." Does that sound right?

Yeah, otherwise the media session spec will break the media parts of the keyboard events spec, even if there isn't an active media session.

from mediasession.

Is it possible to implement that behavior on Android? The documentation says "To receive commands, media keys, and other events a MediaSession.Callback must be set with setCallback(Callback) and setActive(true) must be called."

I haven't tried it, but my guess is that doing that would prevent other applications from receiving media button events. However, if the key events are delivered like any other key events when there is no app currently handling them, that would work.

If the only concern here is existing Web apps, we could of course simply continue to delivering those events when nothing has audio focus, on desktop browsers only. It's not required for compat to use these key events in the new model, if it turns out to be a bad fit.

How should things like onSeekTo(pos) and onSetRating(rating) be represented as a KeyboardEvent? Should we extend KeyboardEvent, even though these events will actually never come from a physical keyboard but rather virtual keys on touch screens?

from mediasession.

I haven't tried it, but my guess is that doing that would prevent other applications from receiving media button events

Oh, if an element in another app/window has media focus, I'd expect only the media-focused element to receive media key events, as if it had a lock on those.

If the only concern here is existing Web apps

Also duplication of listeners. If I'm denied media focus, I should still be able to use the same key events while the window has focus (as long as nothing else on the system has focus).

How should things like onSeekTo(pos) and onSetRating(rating) be represented as a KeyboardEvent?

They don't feel like keys to me, particularly seeking. It'd be great if that could follow the event model of <input type=range>.

from mediasession.

I haven't tried it, but my guess is that doing that would prevent other applications from receiving media button events

Oh, if an element in another app/window has media focus, I'd expect only the media-focused element to receive media key events, as if it had a lock on those.

We'll have to verify how this actually works on Android. I'd be unsurprised if it isn't exactly what would be needed to make the use case work.

If the only concern here is existing Web apps

Also duplication of listeners. If I'm denied media focus, I should still be able to use the same key events while the window has focus (as long as nothing else on the system has focus).

This bit I don't understand, should it be possible to be denied audio focus if there is nothing else playing? Can this happen on any native platform?

How should things like onSeekTo(pos) and onSetRating(rating) be represented as a KeyboardEvent?

They don't feel like keys to me, particularly seeking. It'd be great if that could follow the event model of .

In other words, a simple event named timeupdate or some such, but where should the event handler look to find the new value?

from mediasession.

8b23f34 doesn't use keyboard events, please review.

from mediasession.

See: https://mediasession.spec.whatwg.org/#event-handlers-on-mediasession-objects.

from mediasession.

See: https://mediasession.spec.whatwg.org/#event-handlers-on-mediasession-objects.

Functionality has been moved to https://mediasession.spec.whatwg.org/#media-remote-controls.

from mediasession.

Leaving this to @jakearchibald.

from mediasession.

Was there a reason for not using keyboard events for these events triggered by keys?

The current proposal loses us *up *down events, and event delegation.

from mediasession.

Where would keyboard events be fired, on the active MediaSession object? MediaSession isn't a Node, so there wouldn't be anywhere for the event to bubble.

I'm not entirely certain, but it doesn't look like there are *up and *down events at the Android level for media keys, whether it's the button on the headphone cord or soft keys in a notification.

That being said, on the desktop platforms that currently do fire MediaPlayPause (just Windows?) we need to figure out what to do with it. If we don't fire that event in the same context (Document?) on all platforms, then I think the reasonable thing would be to keep firing these events and if they are not preventDefault()ed let the MediaSession system handle it.

Other ideas?

from mediasession.

Where would keyboard events be fired

On the HTMLMediaElement. That means I can do:

document.addEventListener('keyup', event => {
  switch (event.key) {
    case "MediaPlayPause":
      if (!(event.target instanceof HTMLMediaElement)) {
        // just return, let the default do its thing
        // (the default would be play/pause)
        return;
      }
      event.preventDefault();
      document.querySelector('.first-item-in-playlist').play();
      break;
    // …
  }
});

The above event would handle events that have been targeted to a particular audio/video because of a session, events targeted to a particular audio/video due to tab focus on the element & no media session, and events targeted to an irrelevant element on the page & no media session.

it doesn't look like there are *up and *down events at the Android level for media keys

That's a shame, but I don't think we should limit the web's spec by what the current version of Android does. Of course, the spec should suggest what happens on inputs that do not have up/down support.

from mediasession.

A media session can have more than one participating media element, and eventually likely also Web Audio's AudioContext or AudioDestinationNode. Should the event be fired on all of them, and bubble to Document for HTMLMediaElement but not bubble for AudioContext and AudioDestinationNode?

from mediasession.

Hmm, yeah, it shouldn't fire on multiple elements at once. I guess, even though this a key-triggered system, keyboard events don't fit.

I guess developers will have to add listeners for both media keys, and media session events.

from mediasession.

Before we implement or ship any events on MediaSession we should definitely look into what to do with the existing events. My hope is that there wouldn't be any reason to use them, but if there are ways of using the keyboard events that aren't covered by media sessions, that could point to missing functionality. Getting keyboard events before playback has begun (but without activating a session that could interrupt others) is one such case I'm not sure how to deal with.

from mediasession.

The approach now is that the browser might want to handle some keys. Either virtual or physical and fire the right controls event.

I believe this is resolved. Please reopen if you disagree.

from mediasession.