This is a Unity (2021.3.3f1) Native Plugin to use MediaPipe (0.8.10).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

• Write MediaPipe code in C#.
• Run MediaPipe's official solution on Unity.
• Run your custom Calculator and CalculatorGraph on Unity.
  • ⚠️ Depending on the type of input/output, you may need to write C++ code.

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        var graph = new CalculatorGraph(_ConfigText);
        var poller = graph.AddOutputStreamPoller<string>("out").Value();
        graph.StartRun().AssertOk();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
        }

        graph.CloseInputStream("in").AssertOk();
        var packet = new StringPacket();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone().AssertOk();
    }
}

For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc), so you need to build them first.
For step-by-step guide, please refer to the Installation Guide in Wiki.

⚠️ libraries that can be built differ depending on your environment.

⚠️ GPU mode is not supported on macOS and Windows.

Here is a list of solutions that you can try in the sample app.

🔔 The graphs you can run are not limited to the ones in this list.

Select Mediapipe/Samples/Scenes/Start Scene and play.

If you've built native libraries for CPU (i.e. --desktop cpu), select CPU for inference mode from the Inspector Window.

Make sure that you select GPU for inference mode before building the app, because CPU inference mode is not supported currently.

https://github.com/homuler/MediaPipeUnityPlugin/wiki

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0)
• emscripten (MIT)
  • third_party/mediapipe_emscripten_patch.diff contains code copied from emscripten
• FontAwesome (LICENSE)
  • Sample scenes use Font Awesome fonts

See also Third Party Notices.md.