GDMP is a Godot 3.3+ plugin for utilizing MediaPipe graphs in GDScript.
NOTICE: This branch is for Godot 4.x (GDExtension) and later.
- For Godot 3.x (GDNative) version, please refer to
3.x
branch.
-
Clone the repository, remember to initialize the submodules.
git submodule update --init --recursive
-
Install Bazelisk or Bazel version that meets MediaPipe requirement.
-
Run
setup.py
to generate the godot-cpp bindings as well as apply/setup source code to themediapipe
workspace. Runsetup.py --help
to view the various options.
Example:
./setup.py --godot-binary path/to/godot/binary
- Add calculator dependencies in
GDMP/GDMP.bzl
- Copy
addons
to your Godot project's root directory. - (Recommended) Set
GODOT_PROJECT
environment variable that points to your Godot project, the build script will try to automate copy process if it is detected.
-
Refer to Prerequisite section for Java and Android SDK & NDK setup.
-
Copy or symlink godot-lib to
android/libs
as dependency.godot-lib can be obtained from godotengine.org or from your Godot project's
android/build/libs
if Android build template is installed. -
Run:
build.py android
to build android library, then copy
mediapipe/bazel-bin/GDMP/android/libGDMP.so
andlibopencv_java3.so
toandroid/src/main/jniLibs/arm64-v8a
-
Build GDMP aar with Android Studio or Gradle using the
android
project, then copy the builtrelease
variant aar located inandroid/build/outputs/aar
andandroid/GDMP.gdap
to your Godot project'sandroid/plugins
-
Comment out
macos
andios
part of select() insrcs
anddeps
attributes fromresource_util
inmediapipe/mediapipe/util/BUILD
, this step is required before the ambiguous match problem is solved. -
Use Tulsi project to generate Xcode project.
-
Build
GDMP
static framework target, copy the library from the framework to your Godot project'saddons/GDMP/libs/ios
and rename it with .a extension, so that Godot will treat it like static library instead of dynamic when exporting to Xcode.(Optional) also copy
opencv2.a
to the project and add it as GDNative library dependencies if OpenCV is used in calculators. -
After exporting Godot project to Xcode, add necessary frameworks in order to build the app, and go to
Other Linker Flags
inBuild Settings
to force load the static library, for example:-force_load $(PROJECT_DIR)/path/to/your/GDMP.a
-
Install OpenCV and FFmpeg, then modify
mediapipe/third_party/opencv_linux.BUILD
to make OpenCV visible to Bazel. -
Run:
build.py desktop
to build desktop library, then copy
mediapipe/bazel-bin/GDMP/desktop/libGDMP.so
to your Godot project'saddons/GDMP/libs/x86_64
-
Install MSVC compiler and Windows SDK, then setting
BAZEL_VC
environment variable pointing to your VC installation.Refer to Bazel documentation for more details.
-
Bash is required for building MediaPipe, make sure
bash
is inPATH
or settingBAZEL_SH
environment variable pointing to it.Bash can be installed from Git for Windows or MSYS2
-
Install OpenCV and configure
mediapipe
workspace:- Modify
mediapipe/WORKSPACE
forpath
underwindows_opencv
if OpenCV is not installed onC:\opencv
- Modify
OPENCV_VERSION
inmediapipe/third_party/opencv_windows.BUILD
if OpenCV version is not3.4.10
Refer to MediaPipe documentation for more details.
- Modify
-
Run:
build.py desktop
to build desktop library, then copy
mediapipe/bazel-bin/GDMP/desktop/GDMP.dll
to your Godot project'saddons/GDMP/libs/x86_64
(Optional) also copy
opencv_world3410.dll
to the project and add it as GDNative library dependencies if OpenCV is used in calculators.
-
Go to
Project -> Project settings -> Plugins
to enable GDMP. -
Place files used by MediaPipe in your project's directory, files used by graphs (e.g. TFLite models) need to be placed in your project's directory according to the path provided by the calculator configs.
-
To load a MediaPipe graph:
# load graph config in text or binary format var config: MediaPipeGraphConfig = load("res://path/to/your/graph.pbtxt") # initialize mediapipe graph with graph config var graph: MediaPipeGraph = MediaPipeGraph.new() graph.initialize(config) # initialize GPU resources var gpu_resources: MediaPipeGPUResources = MediaPipeGPUResources.new() graph.set_gpu_resources(gpu_resources)
-
To add a packet callback to the graph's output stream:
graph.add_packet_callback("stream_name", target_object.target_method)
-
To use the packet from graph callback(NormalizedLandmarkList for example):
func _on_new_landmarks(stream_name: String, packet: MediaPipePacket): var landmark_list: MediaPipeProto = packet.get_proto("mediapipe.NormalizedLandmarkList") var landmarks: Array[MediaPipeProto] = landmark_list.get("landmark") for landmark in landmarks: var vector: Vector3 = Vector3(landmark.get("x"), landmark.get("y"), landmark.get("z")) print("x:%f, y:%f, z:%f" % [vector.x, vector.y, vector.z])
-
Start the graph and camera for sending video frames to the graph:
var camera_helper: MediaPipeCameraHelper = MediaPipeCameraHelper.new() camera_helper.new_frame.connect(self._on_camera_new_frame) graph.start() if camera_helper.permission_granted(): camera_helper.start(MediaPipeCameraHelper.FACING_FRONT, Vector2(640, 480)) else: camera_helper.permission_result.connect(self._on_permission_result) camera_helper.request_permission()
-
To handle camera permission result and new frame callback:
func _on_permission_result(granted: bool) -> void: if granted: print("permission granted") camera_helper.start(MediaPipeCameraHelper.FACING_FRONT, Vector2(640, 480)) else: print("permission denied") func _on_camera_new_frame(packet: MediaPipePacket) -> void: graph.add_packet("input_video", packet)
The project is still much work in progress, expect usage API changes in future releases.