Comments (24)
Per https://webkit.org/blog/15063/webkit-features-in-safari-17-4/#source-prioritization it seems this change was shipped in Safari 17.4. Correct? If so, can you clarify what was implemented?
from html.
Yeah, that seems correct. If network conditions are bad the end user won't see media so in practice it prolly doesn't matter, but it seems reasonable to make this more solid, especially as we already have this logic for <track>
.
from html.
@whatwg/media thoughts?
from html.
I think this would require a fundamental change to how the algorithm works. Currently, the algorithm can pick a source
while the parser has yielded before the </video>
tag (so more source
elements can still appear). If the browser needs to have all source
s before making a selection, we need to wait until </video>
is parsed. cc @hsivonen
For the script-created elements case, we can probably do what picture
does: queue a microtask before making a selection.
If the algorithm fails to select something because all type
s are unsupported or all media
s don't match, the networkState
can be NETWORK_NO_SOURCE
and allow for resource selection to happen again if another source
element is inserted (after a microtask).
This change would probably make the spec simpler.
While arbitrary order makes sense for type
, I don't think it makes sense for media
. How do you envision it should work when media
is also used?
Example:
<video>
<source src="a" type="video/mp4" media="(min-width: 600px)">
<source src="b" type="video/mp4">
<source src="c" type="video/webm" media="(min-width: 600px)">
<source src="d" type="video/webm">
</video>
Maybe selection could be in two passes: first, select a list of candidates that has the first (in tree order) source
element where media
matches (whether the attribute is present or not), for each unique type
value. For the above example, the lists would be either ["a", "c"] or ["b", "d"]. Then, select among those in a UA-defined manner, but use the source
element without a type
attribute (if any) as last resort.
from html.
Yeah, I was also thinking this would need to be done in two passes after parsing. And yeah, I was also thinking it would simplify the spec quite a lot, but I was unsure also how much breakage there would be given how much stuff seems to happen right now in the algorithm.
from html.
It seems like the intent of this feature is to allow the browser to make the best choice for users.
In a case like
<video>
<source ...>
<source ...>
[... the server hangs for 5 seconds ...]
<source ...>
</video>
it seems like the best choice is for the browser to give up sometime early during that 5 second window, and choose from the first 2 sources. It would be bad to mandate in the spec that the browser has to wait until it sees the end tag.
Does that fit with what you all are thinking? How does it play into the discussions above about multiple passes after parsing?
from html.
it seems like the best choice is for the browser to give up sometime early during that 5 second window, and choose from the first 2 sources
I don't believe the proposal prevents this behaviour.
If those first two sources aren't playable, then you would need to wait anyway.
If two sources are available and the second is preferable over the first, then the UA play the 2nd.
If there's only one source available at the time you parse it and it is playable you play it.
I would limit to the two passes parsing to only the sources that are immediately available.
The 2nd parse cycle being only required if during the first parse; one of the source was playable, but the system would have liked to see if a preferred one was there (like if the available source only allowed for software decoding).
from html.
As long as this proposal doesn't involve waiting for the end </video>
tag, then that's good. Some of the discussion in #10077 (comment) implied waiting for the </video>
tag, so I wanted to make sure that was not part of the proposal.
from html.
Waiting for the end tag is what I had in mind, to make processing predictable and not depend on network latency. I'm not convinced it's best to allow processing of partial data here. I think speculatively fetching a video is OK, but AFAIK browsers don't do that today.
If the server stalls while parsing the video
start tag, the user also won't see a video, even if the src
attribute has been seen.
But maybe there's some heuristic we can apply to load (and play) sooner without introducing network latency impact on resource selection. For example, when seeing fallback content (non-whitespace text or elements other than source
and track
), run resource selection. A risk here is existing content might have bogus elements or text in video
(between source
elements), which would no longer work, at least if the resource selection only runs once (as today).
from html.
How are you proposing to decide a better source? Do you just want to always select the mp4?
I'm not sure how you'd make a better decision beyond some simple cases (that seem uncommon nowadays) w/o parsing and loading each source -- which could slow down loading significantly. It's not common to list codecs
in type
, so at best you're guessing which codecs and resolution would be in each source and there's significant overlap. E.g., webm -> (av1, vp9, vp8), mp4 -> (av1, vp9, h264, hevc).
from html.
How are you proposing to decide a better source? Do you just want to always select the mp4?
Not at all; we would want to select the "best" source, which could (and has) meant the one with a HW decoder available.
For context, we have previously de-prioritized HEVC on platforms containing only a SW HEVC codec, and way, way, way back in the day, de-prioritized Ogg on systems that had Perian installed as a QuickTime extension.
And of course, this behavior will work better when there is more information provided in type
. But source selection in general works better when there is more information provided in type
.
from html.
Can you clarify "works" in the case of type
w/o codecs
? I don't see how it would even work in that case w/o parsing.
from html.
Adding a <source>
without a type
already requires parsing (in the form of sniffing). This is just a more advanced form of sniffing.
from html.
There might be some privacy issues here. The site could learn more about what sort of hardware the user has.
from html.
There might be some privacy issues here. The site could learn more about what sort of hardware the user has.
MediaCapabilities already expose in much more details the information. Whichever remedial work done for MC is applicable here too (as is, disabling under some circumstances the selection algorithm)
from html.
Interesting, since the privacy and security issues are similar (not the same though) as with delayed clipboard rendering. And that one is being objected strongly because of the privacy issues
from html.
Interesting, since the privacy and security issues are similar (not the same though) as with delayed clipboard rendering.
One exposes specific user behaviour , the other could expose hardware capabilities (information already available). And for the latter we have existing policy to limit exposure covering it
from html.
A possible web compat issue is with error
events on source
elements. Currently, since the source
s are tried in order, sites may use an error
event listener on the last source
element as a signal that all resources have failed to play, and maybe decide to show an error message to the user. The spec even has an example:
https://html.spec.whatwg.org/multipage/embedded-content.html#the-source-element:event-error
If a browser then picks that source
to try first, but it's not playable, it will trigger the error message code path of the web page even though the browser still hasn't tried any other source
.
A possible mitigation could be to defer firing error
events on source
elements until all available options have failed.
from html.
A (too late?) note that from an author education/understanding standpoint, the model for years has been:
srcset
: an unordered list of URLs with descriptors, provided by the author. The UA is best-suited to make decisions about which one should be picked, when, and does so, using the provided descriptors.<source>
: an ordered list of "sources" (each possibly having its ownsrcset
). The author is best-suited to make decisions about which one should be picked, when, and they attach explicit instructions describing the situations in which UAs should pick each.
I agree that for the type-switching use case, UAs will generally make better decisions than authors. The question is less "what does this particular content and page context need" (author best-suited), and more "what does this particular user and UA context need" (UA best-suited).
Rather than changing <video><source>
to have a srcset
-like selection mechanism, it would be much cleaner to introduce srcset
+ a type()
descriptor to <video>
. Teaching and understanding these mechanisms is hard enough, and being able to transfer knowledge about how markup patterns map to mechanisms from one media element to another is great. Muddling mechanisms makes understanding, explaining, testing — authoring — much harder.
from html.
Per https://webkit.org/blog/15063/webkit-features-in-safari-17-4/#source-prioritization it seems this change was shipped in Safari 17.4. Correct? If so, can you clarify what was implemented?
This is the webkit change that implemented the policy on iOS devices only
https://bugs.webkit.org/show_bug.cgi?id=267753
It does as described as an earlier proposal; if at the time of the check we have an alternative source following the one currently being checked, and if the currently checked source is using VP8 (SW only) or doesn't have hardware decoded VP9; then this source will be skipped.
If there's no source to try after, then it will be used.
full WebM support was only added in Safari on iOS 17.4, one of the reason for this decision was to reduce the potential for unexpected regressions.
from html.
(I did some digging and I think that WebKit has had the "wait for end tag" behavior for about 12 years, although seemingly it's only used for <track>
elements. This is the commit I found that seems to introduce that: https://trac.webkit.org/changeset/102968/webkit.)
from html.
Waiting for the end tag for track
s is per spec: https://html.spec.whatwg.org/multipage/media.html#text-track-model:blocked-on-parser
from html.
The code in WebKit to try source
elements is like the spec, i.e. evaluated in order without waiting for </video>
, as far as I can tell.
- https://github.com/WebKit/WebKit/blob/a7b863a49945946c913e6e194ec047da844094a4/Source/WebCore/html/HTMLMediaElement.cpp#L5404 responds to inserting source elements
- https://github.com/WebKit/WebKit/blob/a7b863a49945946c913e6e194ec047da844094a4/Source/WebCore/html/HTMLMediaElement.cpp#L886 ("end tag seen") only invokes text track selection.
It does as described as an earlier proposal; if at the time of the check we have an alternative source following the one currently being checked, and if the currently checked source is using VP8 (SW only) or doesn't have hardware decoded VP9; then this source will be skipped.
If there's no source to try after, then it will be used.
If WebKit doesn't wait for the </video>
end tag, then whether there is an alternative source at the time depends on where the HTML parser yields, which can depend on network conditions, right?
from html.
Note that a server hang already prevents:
track
selectionpicture
selection (it only starts when theimg
is seen)
The effect of not waiting for the end tag is that users will sometimes get a suboptimal format selected, for the same HTML. As I said above, I think processing should be predictable and not depend on network latency.
from html.
Related Issues (20)
- Meeting 2 for joint OpenUI-WHATWG/HTML-CSSWG task force on styleable form controls HOT 4
- Upcoming WHATNOT meeting on 5/23/2024 HOT 6
- [Proposal]: Enable `HTMLElement` attributes to be toggled without JavaScript HOT 2
- Issue with Step 10 of inner navigate event firing algorithm HOT 2
- Provide native validation messages for native validity states on Form Associated Custom Elements
- Upcoming WHATNOT meeting on 5/30/2024 HOT 2
- Feature Request: Top Layer Stack Management HOT 5
- Fetch priority hint for dynamic modules
- a element defines content model as only Transparent HOT 1
- Severing a document's opener relationship regardless of origin HOT 11
- Video loading=lazy attribute support HOT 1
- Specify That Responsive Video Elements Re-Evaluate Sources on Media Change, Like Picture
- HTML Video: Allow a child Picture Element to Control the Video Poster Image HOT 1
- HTML Video and its Source Elements Should Support SRCSET and SIZES
- Upcoming WHATNOT meeting on 6/6/2024
- [Proposal] capture screen
- [Proposal] Capturing Real User Monitoring (RUM) HOT 1
- setTimeout() nesting levels: browsers do not match specification
- Bug in timer initialisation steps: the id's of cleared but still live timers can be reused
- The term "insertion point"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.