Giter Club home page Giter Club logo

microsoft / mixedreality-azurecommunicationservices-sample Goto Github PK

View Code? Open in Web Editor NEW
5.0 5.0 9.0 1.65 MB

Azure Communication Services (ACS) empower developers to seamlessly integrate voice, video, chat, telephony, and email communications into their applications. In this HoloLens 2 sample application, we focus on Teams interoperability, specifically joining Teams meetings.

License: MIT License

C# 98.14% QMake 0.02% ShaderLab 1.84%
azure-communication-services hololens hololens2 mrtk3 teams-interoperability

mixedreality-azurecommunicationservices-sample's Introduction

Sample: Video Calling and Teams in a HoloLens 2 Application

Introduction

This sample demonstrates how to use the Azure Communication Services within an immersive mixed reality application, running on HoloLens 2.

Getting Started

  1. Install Visual Studio 2022
  2. Install Windows 10.0+ SDK
  3. Install Unity 2022 LTS with Universal Windows Platform Build Support
  4. Install .Net 6 SDK
  5. Install git lfs
  6. Clone repository
  7. Access to an Azure Tenant with the necessary privileges for creating new resources and app registrations.
  8. Access to a work/school Teams account with administrative privileges.

Overview

Azure Communication Services (ACS) empower developers to seamlessly integrate voice, video, chat, telephony, and email communications into their applications. In this HoloLens 2 sample application, we focus on Teams interoperability, specifically joining Teams meetings. Here’s what our app demonstrates:

  1. Authenticated Teams Integration: Connect to ACS as an authenticated Teams user using your work/ school credentials.
  2. Microsoft Graph Integration: Retrieve coworker information and upcoming meetings.
  3. Teams Meeting Participation: Join Teams meetings with video and audio.
  4. Live Captions: Render live captions during Teams meetings.
  5. Incoming Calls: Accept 1:1 Teams video and audio calls.
  6. XR Interactions with MRTK3: Utilize the Mixed Reality Toolkit 3 for seamless XR interfaces.

A screenshot of the application's main panel and video call experience. The left  shows a panel with the signed-in user's profile picture and name, along with an upcoming meeting and a list of relevant contacts. The right shows a video panel with call controls.

There are many more Azure Communication Services features not covered by this sample application. Please visit the Azure Communication Services documentation for additional features and pricing.

App Components

This immersive HoloLens 2 sample is made up of two parts:

  • Unity App. A native application that runs on HoloLens 2 devices.
  • Web App. An Azure Function application that returns an Azure Communication Services access tokens for an authenticated user using a Microsoft Entra ID.

The following Microsoft components are used by this sample:

Block diagram detailing the components used in this sample application

Next Step

The next step, Setting Up Azure Communication Services, will walk you through how to create an Azure Communication Services resource.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

mixedreality-azurecommunicationservices-sample's People

Contributors

marlenaklein-msft avatar thv2022 avatar amollis avatar microsoftopensource avatar dependabot[bot] avatar microsoft-github-operations[bot] avatar

Stargazers

Futo Horio avatar  avatar  avatar sara avatar  avatar

Watchers

James Cloos avatar .NET Foundation Contribution License Agreements avatar Vanessa Oliva avatar  avatar  avatar

mixedreality-azurecommunicationservices-sample's Issues

Meeting UX - Caption UI

-Captions are a little small / squished and may be hard to read.
-Close caption below video window is stretched differently if the person speaking has their camera on.

MSAL auth on device

While MSAL auth works in editor, it does not work in device.
WAM is recommended for device, but it may be desired to use MSAL.

UX - Meeting bar is too low

The meeting bar is too low and not in the field of view.

The control bar and the image of other attendees should be closer together.

Change feature not supported message

Change all feature not supported messages ("This feature is currently not supported by Microsoft Graph, implementation can be found in code") to:

For 1:1 Calling: "This feature is currently not supported by ACS."
For other unsupported features (adding participants, add/removing attendees): "This feature is not yet implemented."

UX - Search functionality

When typing the first letters of somebody's name, there is no indication that a search is happening.

Clean up missing scripts and prefabs

  • Remove or replace the missing scripts and events on PeopleGetter and PeopleSearcher.
  • Remove or replace missing join call event and missing Participant Prefab on MeetingManager.
  • Remove or replace the CenterContent prefab

image

image

image

image

Control bar should disappear when removed from meeting

Behavior: In a HoloLens to HoloLens call, when a participant is removed during a meeting or call, the removed HoloLens participant can still see the control bar. The removed HoloLens participant must click the leave button to return to the main panel.

UX - Login Panel

The Login Panel should follow you as the other panels do and should have a pin option.

UX - Caption Box

The caption box gets weirdly warped.
It should not be able to flip backwards or change the z-dimension.
image

It also breaks apart sometimes.

UX - Text

The text is difficult to read, especially when moving.

Meeting UX - Button Visuals

“People” in call icon isn’t centered.

image

There's a blank area at the bottom of the meeting title. Either use the space (i.e. show who's currently speaking) or format similar to dashboard UX.

image
image

Enable SpeechInteractor

In order for see-it say it labels to appear and voice commands to work, the SpeechInteractor must be enabled.
Verify that this does not interfere with meeting audio and that voice commands work to select buttons.

UX - More obvious buttons

Make it more obvious when the camera, audio, mic is on and off (different colors?).
Make the join button more obvious (green).

Dashboard UX - z-axis rotation

All window manipulation doesn’t need to rotate in z axis. This will tilt the window causing text to be hard to read.

Remove z rotation in window manipulation, for both dashboard and in-meeting UX panels.

Polish Participants Panel List

Tasks

Document Known Issues

Add documentation for known issues:

  • Other users need to toggle camera off and on before the ACS user on HL2 can see it (ACS bug).
  • Auto sign-in does not work (doesn't grab tokens correctly / cannot join meeting.)
  • Doesn't decline other calls when in a meeting
  • Bar doesn't disappear when removed from meeting
  • MSAL auth does not work on device
  • Audio and camera do not work via remoting

Tasks

Prevent console errors

Several console errors appear consistently:

"[PhotoGetter] Failed load image data into 2D texture" when first logging in.

and

"IndexOutOfRangeException: Index was outside the bounds of the array. ParticipantVideoController.SetIconVisibility (System.Boolean visible) (at Assets/Scripts/UI/ParticipantVideoController.cs:234)"
"Error HangUpCurrentCall. Exception: Unknown error. UnityEngine.Debug:LogError (object) CallScenario/d__91:MoveNext () (at Assets/Scripts/Calling/CallScenario.cs:465)"

whenever a meeting is exited.

Errors should be investigated and avoided if possible.

UX - Recent contacts spacing

Validate cases with many recent contacts (more than would fit in the space).
There is a gap in the recent contacts box which maybe should be removed.

Implement auto-sign in

Implement the auto sign-in option so that it grabs tokens properly and the user can join meetings.
If this is not possible, remove the option for auto sign-in.

UX - Update login App Name

Update the app name on the login screen to "ACS Showcase Sample App" from "ACS HL2 Sample App". The logo image should be changed.

image

Decline other calls when in a meeting

Behavior: When a user receives an incoming call while in another call or meeting, the incoming call is not immediately declined; the call rings for the user making the call until it times out.

UX - Broken toggle for relevant contacts

When selecting a relevant contact, the profile will toggle to the call image. It does not clear when closing the feature-not-implemented popup. When selecting the profile again, it will toggle back to normal and the pop-up will appear again.

Since the feature is not implemented, once the pop-up is closed, the profile image should toggle back to normal.

Dashboard UX - Delayed head tracking

When the dashboard is unpinned, the loose head leash is too slow, it needs smoothing on entry and to be a bit quicker. Most of the time, we look away and wonder if the UI will show up.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.