kixelated / moq-js Goto Github PK

The current code is a player only. We need to support contribution, where we use WebCodecs to encode media and send it over the network. I started but didn't get very far.

We should add support for Firefox once they have WebCodecs support. I'm not sure when that will arrive: mozilla/standards-positions#209

I removed this functionality as part of migrating away from ANNOUNCE. Time to add it back and do it properly this time!

1. Add a "public" checkbox to /publish that allows the user to enter the title of the broadcast.
   • The title is not the same as the random broadcast ID.
   • The broadcast ID is random and used by relays (moq-api) for routing.
   • The broadcast title is human-readable, only used when the publisher wants to make a public broadcast.
2. Add an endpoint that creates a new broadcast given the form information. It should:
   • Verify the information: title, codec info, etc.
   • Generate random broadcast ID.
   • (future) Generate an auth token tied to the broadcast ID.
   • CHOICE: Choose the database. We're running in GCP so there's quite a few options, but local development should still work.
   • Insert everything into the database.
   • Return a URL used to connect to moq-relay: ie. https://relay.quic.video/<broadcast ID>
   • CHOICE: This endpoint could be part of moq-api or moq-js; your choice.
3. Add an endpoint that lists all PUBLIC broadcasts.
   • Periodically verify each broadcast is still live via moq-api.
   • CHOICE: This endpoint could be rendered on the client or server side (moq-js only).
   • Return a paginated list of public broadcasts and some basic information about them.
4. Render the list of all public broadcasts on /watch.
   • Make it pretty.

In moq-pub, it's possible that the bitrate is not passed in the track data:
https://github.com/kixelated/moq-rs/blob/75e7dc03bfc0e64fe243742478bd755b399045a2/moq-pub/src/media.rs#L188

I'm using dev/pub to test the player and changing it to try out something. I needed to use Catalog.isVideoTrack but it expects a video track to have frame_rate:

I think we need to remove this check.

Screen Capture API can be used.

https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture

@kixelated
A drop-down that lets you switch between them on the playback side
that then leads directly into ABR
and screen sharing
on the publish side, we gotta decide if we send tracks in parallel or separately
and how do we flag that we're now transmitting a separate track, so the player needs to change
(probably via catalog updates)

The current visualizer only shows the min-max range since I was lazy. It should be able to render holes but we need to return them within the player API.

It's currently not possible to pause playback. There should be a pause button somewhere, or it occurs when you click on the window.

We currently only support h.264 and AAC because I'm too lazy to test anything else. It should be easy enough to configure WebCodecs with the correct MP4 atoms.

You can’t publish/subscribe to optional tracks, which is required for any form of ABR.

Testing under Chrome 117.0.5938.88, mac 13.5.2, the quic.video/publish UI does not enumerate either of my video sources. Screen shot

Here is https://webrtc.github.io/samples/src/content/devices/input-output/ for comparison

The backup MSE player sometimes hangs and I'm not sure why. It could use some love.

I disabled audio because I was getting tired of working on the player. It needs to be synchronized with video, which is actually kind of annoying thanks to WebAudio.

It's currently not possible to change the volume. This is not a big deal while audio is broken... but will be when #3 is fixed.

The current player keeps going if there's a media underflow. This is kind of hilarious because the player keeps going even when the stream is stopped, but it should pause and possibly increase the buffer size instead.

Add support for https://afrind.github.io/draft-frindell-moq-chat/draft-frindell-moq-chat.html

You can’t add/remove tracks once the broadcast has started, such as to change input device or screen share.

We need to support bidirectional broadcasts and rooms, which means echo cancellation.

Audio and video are rendered out of sync. This isn't a big deal with zero latency, but matters more at higher end targets.

We should visualize the available tracks and which ones are active.

Media should be prioritized by the sender during congestion. Use priority hints when implemented.

I run moq-cert, moq-relay, moq-pub, moq-js on ubuntu 22.04
then I access https://localhost:4321/watch//?server=localhost:4443 with chrome 117.0.5938.149
no network error reported in terminal, but I failed to play the video
the website shows this: WebTransportError: Connection lost
can I get some help?

Currently, WebTransport in Chrome does not accept self-signed certificates, even if they're from a local root. The work-around is to use the serverCertificateHash field when establishing the connection to explicitly list which certs are accepted. The certificate MUST use ECDSA and be valid for <14 days.

This is really annoying. The current approach is to host a /fingerprint endpoint on the server. The client issues a HTTPS request (yes, using the same cert) to get the cert before then connecting via WebTransport. It doubles the TLS handshakes and round trips. It also means the certificate has to be regenerated every week or two.

I was thinking we should render frames in the buffer, instead of showing solid bars only. It would be really cool to see some video frames in the timeline. I was thinking we could render the most recently received frame(s) with a box around it to really drive home that media can arrive out of order.

Just parsing the catalog data is wrong here.

If I add a new method to the Catalog class and call this method on a newly created Catalog instance, the runtime gives an error (method not found).

Instead changing the code as follows is better IMHO:

const catalog = new Catalog()
catalog.tracks = JSON.parse(str).tracks

Heya,

Your blog is great, I love it. Can you add an Atom or RSS feed please?

We should add support for Safari once WebTransport support is added: WebKit/standards-positions#18 (comment)

Reno/CUBIC (default) suffers from buffer bloat. Use congestion control hints when implemented.

Needs to be done in tandem with moq-rs.