Giter Club home page Giter Club logo

opentera-webrtc's Introduction

OpenTera - WebRTC

Actions Status

WebRTC is a standard for real-time audio/video/data communication and is mostly used in web browsers. For the OpenTera micro-service architecture, we wanted to use Google's native WebRTC implementation for robots so we would have more control on the streams and develop a framework that is optimized for robot teleoperation. We also wanted to use hardware acceleration if possible on embedded platforms.

This project provides:

License

By default, libwebrtc is built with non-free codecs. See webrtc-native-build to build without them.

API Documentation

API Documentation is available here.

How to Contribute

Please read the Code of Conduct and CONTRIBUTING files.

Dependencies

Build Instructions

Examples

C++

Python

Javascript

Hardware Acceleration with GStreamer

Hardware acceleration depends on the platform you are using to build opentera-webrtc. The following table summarizes supported hardware and GStreamer plugins:

Platform VP8 VP9 H.264
Jetson TX2/Nano nvv4l2vp8enc, nvv4l2decoder nvv4l2decoder nvv4l2h264enc, nvv4l2decoder
Jetson Xavier NX X nvv4l2decoder nvv4l2h264enc, nvv4l2decoder
Jetson AGX Xavier X nvv4l2decoder nvv4l2h264enc, nvv4l2decoder
Jetson Orin / Orin Nano / Orin NX X nvv4l2decoder nvv4l2h264enc, nvv4l2decoder
Raspberry Pi 4 X X v4l2h264enc, v4l2h264dec
VA-API vaapivp8enc, vaapivp8dec vaapivp9dec vaapih264enc, vaapih264dec
Apple Media X X vtenc_h264, vtdec

Note that VP9 encoding is under development, see issue 117.

ROS

  • The C++ library and Python bindings are used in our opentera_webrtc_ros ROS package. RGB images published in ROS topics can be easily used for streaming in a RTCPeerConnection using our signaling server. Hardware acceleration and codec selection can be used if available.

Frontend

Authors

  • Marc-Antoine Maheux (@mamaheux)
  • Dominic Létourneau (@doumdi)
  • Philippe Warren (@philippewarren)
  • François Michaud (@michaudf)

Contributors

  • Cédric Godin (@godced)
  • Ian-Mathieu Joly (@joli-1801)

Sponsor

IntRoLab

IntRoLab - Intelligent / Interactive / Integrated / Interdisciplinary Robot Lab

opentera-webrtc's People

Contributors

dependabot[bot] avatar doumdi avatar g-lauz avatar joli-1801 avatar mamaheux avatar philippewarren avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

pandinosaurus

opentera-webrtc's Issues

Native Android support

I tried to build an android project using C++ but found that the library does not support building under the NDK

Integrate GStreamer Encoders and Decoders

https://opensource.apple.com/source/WebCore/WebCore-7609.1.20.0.9/platform/mediastream/libwebrtc/

  • CMake option to use GStreamer or not
  • Add a parameter to force the hardware codec in the SignalingClient constructor (optional)
  • The encoder factory should try to use the hardware accelerated encoders, then fallback to the default webrtc-native ones. The GStreamer encoder class can use encodebin. A GStreamer encoder can be prioritized by changing the rank.
    • nvv4l2h264enc, nvv4l2vp9enc, nvv4l2av1enc for Jetson platforms
    • vaapih264enc, vaapivp8enc, vaapivp9enc for Intel CPUs and some AMD Radeon
    • v4l2h264enc for Raspberry Pi platforms
  • The decoder factory should try to use the hardware accelerated decoders, then fallback to the default webrtc-native ones. The GStreamer decoder class can use decodebin. A GStreamer decoder can be prioritized by changing the rank.
    • nvv4l2decoder for Jetson platforms
    • vaapih264dec, vaapivp8dec and vaapivp9dec for Intel CPUs and some AMD Radeon
    • v4l2h264dec for Raspberry Pi platforms
  • Add the GStreamer support in the root README.md

Compilation fails on OpenCV HighGUI

Hey,
I am trying to compile the code on Ubuntu 20.04, I have cloned the code, pulled the submodules then followed the instructions, during the make command I get the following error:
image

I assume the build should be self contained, so what I am missing ?

Write Readme

Add description and documentation to the readme file.

Fix GStreamer VP9 encoding

When enabling software GStreamer video codecs and forcing VP9, the video streams do not works.

How to reproduce

  1. Modify the CppStreamClient example to use software GStreamer video codecs and force VP9.
  2. Execute the WebStreamClient example.
  3. Execute the CppStreamClient example.
  4. Perform a call.

Unreliable tests on macOS that sometimes fail unexplicably

Describe the bug
Some of the tests on macOS are flaky (they sometimes work, and they sometimes fail unexplicably).
Frequently caused by sockets (socket.io) connections.
Also sometimes it is because of broken pipes (SIGPIPE).
Also sometimes, tests just fail.
This does not happen in GitHub's CI macOS runners, only on local machines.
There might be two different problems: websockets (socket.io) and datagrams (low-level libwebrtc).
The socket function fails inside of libwebrtc (here).

To Reproduce
Run tests on macOS multiple times.

Expected behavior
The tests always succeed.

Desktop (please complete the following information):

  • OS: macOS
  • Browser: N.A.
  • Version: N.A.

Additional context
This could be caused by a security feature of macOS 13, maybe. This is worth exploring.

Add mute and close/open camera features.

In the web client it's relatively easy to open and close some track of the stream using MediaStream and MediaStreamTrack :

let localstream = null;
openteraWebrtcWebClient.devices.getDefaultStream().then(stream => {
    localStream = stream;
});

// Mute mic
localstream.getAudioTracks().forEach(track => track.enabled = false);
// Close camera:
localstream.getVideoTracks().forEach(track => track.enabled = false);

A set of function like mute(true) and cameraOpen(true) could be useful particularly for the C++ and the Python client where the implementation of this features could be more complexe.

Pybind11 >2.6.0 blocking

  • Python tests on version 3.6, 3.8 do not work.
  • FunctionTask::callSync seems to block. We should investigate the underling webrtc thread. Lambda functions are never called.

Add AudioStream support in library

Audio is required for bi-directional webrtc communications in some use cases. This would allow to use ODAS separated audio directly as an audio input stream. Other applications would include speech-to-text features where an audio stream gets processed and returns corresponding text in the data channel.

Log a warning when the user register a null value as localStream (opentera-webrtc-web-client)

When the user instantiate a new StreamClient he can pass a null value in the attribute localStream of the streamConfiguration parameters of the constructor:

const StreamConfiguration = {
    localStream: nullSrcVideoElement.srcObject, // Where nullSrcVideoElement.srcObject = null
    isSendOnly: false
};
streamClient = new openteraWebrtcWebClient.StreamClient(SignalingServerConfiguration, StreamConfiguration, RtcConfiguration, logger);

If the user expect that he had pass a valid value to localStream it can be really hard for him to figure it out what's wrong in his code without any warning.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.