When and how should we load media metadata artwork? about mediasession HOT 28 CLOSED

Right now the spec does not say we must wait for the fetch steps to complete before showing media controls.

I agree it could be good to have a preloaded image. Perhaps we should define 'display media controls' steps similar to notification's display steps that wait until any related fetch steps have completed.

Alternatively, maybe it is ok to 'lazy load' artwork URLs so the fetch steps do not block any display steps.

WDYT?

from mediasession.

I can imagine being okay with showing the artwork at a later point (if you're bandwidth constrained, or the user skips very quickly, or some such), but I think by default you want to show the correct artwork and the API needs to accommodate for that.

An alternative could be that you pass a Blob rather than a URL, but maybe that's too low-level... If you stick with URLs though I think you might want either events or some kind of property that returns a promise when everything is loaded.

And the examples should indicate that while you play the current song, you're already caching bits for the next song. And that once you start playing the next song, enough is cached to show the image and play it. (And then you start caching for the song after that of course, or some such.)

from mediasession.

setMetadata itself could return a promise that resolves when the artwork is loaded. Not sure if we'll regret that if we end up with more than one kind of artwork, though.

from mediasession.

It seems setMetadata() is the wrong API, since you might end up changing title/etc. while still showing the old artwork?

from mediasession.

Perhaps as with <canvas> we can require one of HTMLImageElement & friends?

from mediasession.

Is ImageBitmap a good primitive for an already-loaded-and-decoded image? We've talked about that, but it seems annoying if all you have is a URL, and what about cross-origin artwork?

from mediasession.

Passing a URL might still be okay, but in that case you need a distinct object I think that holds all the metadata and is responsible for loading assets. And then separately you can decide when that object is going to take over from what is displayed now.

[Constructor(MediaSessionDataInit init)]
interface MediaSessionData {
  ...
  readonly attribute Promise<boolean> done;
};
dictionary MediaSessionDataInit {
  ...
};
partial interface MediaSession {
  void setSessionData(MediaSessionData data);
};

from mediasession.

Yeah, something like that could work. We had talked about having a mutable MediaSessionData object on the MediaSession object, but that would amount to a tear-off and for any mutation one would have to enqueue a microtask to actually use the new state to group all changes in a script run. Having a separate constructible object with readonly members doesn't have those problems.

from mediasession.

This issue was originally discussed in #44.

One way a web developer could ensure media artwork loads immediately is to pass a pre-fetched Blob or data URI as the 'artwork URL'.

Media artwork URL fetching is (currently) unobservable from a web page so perhaps we could defer to the implementation stage to figure out exactly when this fetch should happen. It feels like we can have interoperability in the JS platform without initially specifying this and there are different fetch strategies that could work quite well here.

I could be wrong :/

from mediasession.

We should not rely on fetching blob or data URLs (especially the former) to happen synchronously.

from mediasession.

We should not rely on fetching blob or data URLs (especially the former) to happen synchronously.

Right. I also plan to remove invocation of the fetch steps from the setMetadata(metadata) algorithm.

At this point it's unclear if we need to change the API. If we want to discuss changing setMetadata to something else perhaps we should re-open #44?

from mediasession.

If you remove the fetch steps, when would the image be fetched?

from mediasession.

I've been trying to prototype a solution for this. First, a URL and Promise pair:

partial interface MediaMetadata {
  attribute USVString artwork;
  readonly attribute Promise<Response> artworkLoaded;
}

Setting artwork starts the fetch, and artworkLoaded is the promise returned by the internal fetch(). I'm not a huge fan, it's kind of duplicating fetch() and doing metadata.artworkLoaded.then(...) before setting metadata.artwork wouldn't work.

Another idea, not implemented, and possibly crazy:

partial interface MediaMetadata {
  attribute Promise<Response> artwork;
}

The API would then be metadata.artwork = fetch(artworkURL), which looks OK, but no existing API works like this. I think making the type simply Response would not work, because then one cannot start the fetch at the latest opportunity, which a simple USVString does allow.

@annevk, advice on how to integrate with Fetch much appreciated!

from mediasession.

We need to resolve this in order to put artwork back in the spec. Currently, I'm thinking that the MediaMetadata constructor shouldn't fetch, but rather the fetch steps (currently never invoked) should be run when the browser knows it's going to use the artwork.

But, how do we make prefetching the artwork load? The best I can come up with is a fetchArtwork() method, but there's something not-great about integrating with Fetch by adding a new method that's a specialized variant of fetch().

@annevk, thoughts on Fetch integration would be much appreciated. At one point I was thinking that the artwork attribute should be of type Promise<Response>, but that would be a first for the platform I think.

from mediasession.

@annevk replied (I asked on the wrong issue at first):

For notifications we just fetch as a side effect of the constructor/factory method. That's not super clean. A promise for a Response object does not seem entirely unreasonable. I guess it depends a bit as to what you're aiming for. Looking at the latest in <canvas> might also be an idea, by just feeding artwork one of the various ways to represent an image.

from mediasession.

The simple options when going with artwork as a URL are:

1. Fetch in the MediaMetadata constructor
2. Fetch when setting the session.metadata attribute
3. Don't define exactly when to Fetch

from mediasession.

Looking at the latest in <canvas> might also be an idea, by just feeding artwork one of the various ways to represent an image.

Do you mean CanvasImageSource? I'm thinking that just for notification icons, it's likely that media session artwork will evolve to allow some kind of manifest for different screen densities and similar, could that be done by pointing to an img element that's the child of a picture element?

from mediasession.

img elements are probably not a good fit as the artwork will be rendered outside the normal layout tree. I guess the simplest here would be to do something similar to notifications, which is also a little vague as to when fetching happens.

Did you check what iOS and Android do?

from mediasession.

Yes, @doomdavve took a close look at the iOS and Android APIs when we were starting this. IIRC, both allow you to either give a URL or a handle of sorts to a decoded image. We'll be implementing it by passing a decoded image, so that gives us rather a lot of freedom in how the web-exposed bits look.

I'm leaning towards just going with an URL for now and not defining when the fetch steps are invoked, just what they do. Do you think it would be preposterous to later add fetchArtwork() or similar for preloading, or is there just no master plan for how to do preloading using Fetch for the gazillion APIs that currently use URLs only?

from mediasession.

I think for preloading you'd use https://w3c.github.io/resource-hints/.

from mediasession.

Thanks, I knew that was a thing but it still didn't come to mind! There's also https://w3c.github.io/preload/ and it looks like something like this might work:

var metadata = new MediaMetadata({ artwork: "artwork.jpg" });
var res = document.createElement("link");
res.rel = "preload";
res.as = "image";
res.href = metadata.artwork;
document.head.appendChild(res);

I'm unsure if getting the referrer right might ruin it, @richtr's spec text that I've revived says "no referrer" but I assume that the above would send a referrer. Is there a way to inhibit that? The referrerpolicy isn't on HTMLLinkElement in https://w3c.github.io/webappsec-referrer-policy/ and I guess there's a reason. @mikewest @jeisinger?

from mediasession.

Hmm, per w3c/webappsec-referrer-policy#15 that should have been fixed.

from mediasession.

right, however, it's not yet implemented nor shipped in chrome

from mediasession.

OK, sounds like the right primitives are in the pipeline, so for #126 I think I'll keep the "fetch steps" as late as possible, and lean on preloading to handle this. Thanks all!

from mediasession.

FWIW, Android doesn't quite work well with URLs for notifications.

IMO, it would be better to let the UA decide when to fetch. Ideally, if the web application cares about the artwork being available quickly, it could download it and cache it in a SW so it would work offline and be quick to grab on demand.

from mediasession.

FWIW, Android doesn't quite work well with URLs for notifications.

Is this only because the resource is needed in many different sizes, or is the timing of the fetch also a problem?

from mediasession.

Hmm, I should have said "Android APIs". There are these METADATA_KEY_* fields that one can use to set up some info. I was told by some Android folks that the METADA_KEY_*_URI ones are not used by some components like Wear. As a result, we must use the Bitmap ones. It's also better for the UA to control the fetch instead of hand it off to the system because it allows SW to intercept it.

from mediasession.

@doomdavve has a WIP prototype adding artwork to MediaMetadata where the URL is fetched and decoded as an image similar to the implementation for Notifications. So then one hands a Bitmap to the Android APIs and that seems to work well.

from mediasession.