livepeer / go-livepeer Goto Github PK

Add a Relayer struct in BasicNetwork to relay on the messages.

Related to protocol issue - livepeer/protocol#40

The change should be reflected in eth client, and in our job claim verify loop.

Eth client tests require Geth to be running. We may want to skip it for normal continuous integration processes.

This link mentions how to skip tests - https://splice.com/blog/lesser-known-features-go-test/

The broadcaster should keep track of all the transcoding result by creating a new stream that contains a master playlist.

This new stream will be used when viewers request for the ABS stream.

Someone can potential re-use the same priv/pub keypair to start multiple nodes. What's stopping them from doing such?

Do we have to do a seek for the node's ID at the bootstrap process to make sure no node with the same ID already exists on the network?

The Livepeer network should be able to output a live stream into a storage solution.

Currently the best way to do it is using the broadcasting software (for example, OBS allows you to broadcast and record at the same time).

But we should also plan to integrate with cloud solutions like Swarm, IPFS, and S3.

We want the JobID of the newly created job, doesn't seem to be in the data field.

BasicStream sometimes will give an error EOF.

I suspect this is due to 2 go routines writing and reading the same stream at the same time.

Need to do more testing to figure out what's going on.

When the broadcaster receives a video stream, create a on-chain transcoding job with a desired video profile.

Utilize puppeth

Should check Gas after calling Claim and Verify to make sure the transaction was successful.

Add a -transcode=true flag for cli, and kick off a monitoring process when it's passed in.

Broadcaster should verify the transcoder identity after receiving the transcoded video streamIDs

Should be able to load keys from datadir instead of generating new keys every time.

Right now messages can get dropped by the networking layer if it's coming too quickly. (See example in basic_network_test.go/TestSubscriber). We could make it faster by spinning off go routines for message handling instead of doing it in a loop.

ffplay skips due to the initial delay when video is requested from the network.

A potential fix is to change the logic in livepeer stream, where we request the video first, and don't invoke ffplay until we know the video started on the local node.

Need to create a new message type to the protocol so the broadcaster can be notified when the transcoding job is accepted by a transcoder.

related to livepeer/protocol#38

Integrate go-ppspp.

Tasks:

• Separate Livepeer network p2p protocol from video delivery (ppspp) protocol - this means each will have its own handler
• Implement the VideoNetwork interface

When broadcasting finishes, the state needs to be cleaned up. The broadcaster streamDB needs to know which streams the original HLS stream maps to.

The node should have a native go API package. This can then be utilized by the http & ipc packages for exposing the API.

Add a Transcode Job, following the workflow diagram

The transcoding job does the following:

• Subscribes to the broadcasted stream
• Creates new streams based on the transcoding config
• Make the new streams known to the broadcaster.
• Does the transcoding operation on each chunk as soon as it becomes available
• Populate the new streams with transcoded chunks.

Subscribe to a HLS stream, kill the player, wait for the unsubscribe worker to fire, node crashes.

This is to ensure the transcoder is running the correct version of ffmpeg on the correct platform. It's important to ensure the transcoder is running the same software as the verifiers, so the output will be able to match in a deterministic way.

This should be an easy challenge/response style workflow - so that when the transcoder comes online, the network is fairly certain there wouldn't be an issue during verification.

On the old Toynet we had a CDN sitting on top of a public gateway for consuming the streams by ID from our gateway nodes. It would be great to set this up again on the new node.

Get broadcaster and subscriber working for basic network

We should be able to peak into what's happening on a Livepeer node. Information we would like to have:

• Network info (# of peers, # of active streams, bandwidth usage, current network interfaces)
• Video info (Broadcasting videos, all the video streams, all the video buffers, transcoding jobs)
• Eth info (Transcoder stake, transcoder stats, token balance, current round, current cycle)

Sometimes the segmenter times out, but the RTMP connection seems to be still up. The broadcaster now has no idea the stream is no longer working anymore.

We should have a graceful fallback strategy, possibly try to re-create the stream if the RTMP stream is still coming in.

This is because we are not cleaning up the state in the subscriber correctly.

To reproduce:

• set up a broadcaster and a subscriber
• view the video through the subscriber by using ffplay
• kill ffplay
• try to view again
• get a 500 error

Description: A description of your project follows. A good description is clear, short, and to the point. Describe the importance of your project, and what it does.

Table of Contents: Optionally, include a table of contents in order to allow other people to quickly navigate especially long or detailed READMEs.

Installation: Installation is the next section in an effective README. Tell other users how to install your project locally. Optionally, include a gif to make the process even more clear for other people.

Usage: The next section is usage, in which you instruct other people on how to use your project after they’ve installed it. This would also be a good place to include screenshots of your project in action.

Contributing: Larger projects often have sections on contributing to their project, in which contribution instructions are outlined. Sometimes, this is a separate file. If you have specific contribution preferences, explain them so that other developers know how to best contribute to your work. To learn more about how to help others contribute, check out the guide for (setting guidelines for repository contributors)[https://help.github.com/articles/setting-guidelines-for-repository-contributors/].

Credits: Include a section for credits in order to highlight and link to the authors of your project.

License: Finally, include a section for the license of your project. For more information on choosing a license, check out GitHub’s licensing guide!

bandwidth info is needed to create master-playlist on the broadcaster.

The Livepeer daemon should be able to communicate with other local processes through IPC.

Currently the broadcaster publishes a pretty big video. We should make it smaller.

Need a better way to handle the eth password. Take a look at unlock account in Geth.

Transcoders should be able to validate they are running the correct version of FFMPEG, and the output is "to-spec" with the protocol.

The naive strategy here would be to have a few test video segment input/output pairs as "gold standard", and the transcoders would run transcoding on those to make sure they are using the right software.

This is NOT a bullet-proof solution. I'm interested in hearing other ideas.

Can be a very basic smart contract + command line tool right now, the point is to have an easy way to get some test Livepeer token.

A video subscriber should check for the signature when receiving the video chunks. This applies to the transcoder and anyone who is viewing a video stream.

To support ABS, broadcaster should keep track of all the transcoded streams, and when asked, return a master playlist with all the streams / bitrate information. This requires a new network message type.

During testing, we'll get this error sometimes: exceeds block gas limit, especially when calling verify() very rapidly.

How do we adjust the gas per transaction in this case?

The HLSMux, Stream, and Subscriber stays around after the stream ends (receiving a FinishStreamMsg). We should clean it up.

This is non-trivial because we don't know if the stream exists somewhere else in the network. Of course, a simple solution would be to remember the ended stream state for some time, and automatically refuse connection if it's requested.

The mediaserver should ask the network for a master playlist when receiving a request and there isn't a local master playlist already.

It should also return a special error if it detects the request is for a media-playlist that's contained in a local master-playlist. This way we don't have to go to the network, and the mediaserver will know to ask for the media-playlist.

We already have the mediaserver handling video. Now we just need to send it into the network layer to it's transferred over the wire.

Make sure videos can be consumed from a centralized gateway node / page

Open source the player page so anyone can run a gateway node / page

Add a paragraph in the tutorial on how to set up a gateway node / page

The visualization tool is a little out of date. We should update the instrumentation for critical events.

To support ABS, broadcaster should keep track of all the transcoded streams, and when asked, return a master playlist with all the streams / bitrate information. This requires a new network message type.

We want to keep track of the peers, especially when we lose connectivity.

Response from the libp2p team:

For this, you can create a network notifee that receives notifications about peers connecting and disconnecting to your node. The notifee interface is here: https://github.com/libp2p/go-libp2p-net/blob/master/interface.go#L156 And then to subscribe for those notifications, call swarm.Notify(nn) (https://github.com/libp2p/go-libp2p-swarm/blob/master/swarm.go#L355). With that, you can implement your own logic on managing the number of peer connections you have. In ipfs right now we don't yet set any limits with regards to number of connections as we havent determined a good heuristic on what a 'useful' peer is. We are going to be working on that for ipfs pretty soon, but you should be able to implement such logic for yourself pretty easily.

We can also keep track of the bandwidth of the swarm by use a "bandwidth metrics collector" when creating the swarm. https://github.com/libp2p/go-libp2p-swarm

A node has to have a Eth account.

The node should take

• a -transcoder flag to declare that it wants to be a transcoder
• a -maxprice flag to declare the advertised max price
• a -gethipc flag to specify where the geth.ipc file is

If the node is started with -transcoder=true flag, then it monitors the local ethereum node for transcoding jobs. It starts transcoding when a job is assigned to it.