Hey there, thanks for the amazing job here! I wonder if you have plans to release a UE5.1.1 version, so...
Would it be possible to release a compiled version with the current UE5.1.1? Thanks in advance!

hi i'm trying to puppeteering avatar using mediapipe in unity. but it's quite hard.

will you opensource this awesome project later?
or could you elaborate how did you get local rotation from mediapipe pose landmark?? it's too hard for implementing hands and pose ik solver...

          1. Regarding the MediaPipe error, post the log file at here when an error occurs, which is usually in the saved/logs folder so that I can find out what the problem is.

2. Network, I don't think multiplayer animation has much to do with MediaPipe4U, it should be a server streaming animation data and client-side display problem, I believe UnrealEngine has the example of that. You can find UE document or market to get the sample.

3. Important note about the MediaPipeAminInstance (your animation blueprint). At the same time in the world outline you must make sure it has only one MediaPipeAminInstance , If you want to sync animation to many actors, you can use IK Reatarget to anothers. Because the landmarks data of mediapipe that made by MediaPipeHolisticComponent can be deal one time, multiple consumers (MediaPipeAminInstance ) will be get a low animation frame rate (Multiple MediaPipeAminInstance will be competition for data).

BTW: sorry for my bad English , this is a question of the structure of the program, which is difficult to explain in a short answer.

Originally posted by @endink in #20 (comment)

MediaPipe4U Version

20230309

UE Version

5.1.x

UnrealEngine Type

Install From Epic Games Launcher

What happened?

Great project! I downloaded and installed the Mediapipe-4u-Demoand it worked fine for a while when using a webcam then stopped working. The logs had the following errors:

LogMediaPipe: Error: MediaPipe4U license has expired !
LogMediaPipeNvAR: Error: [NvAR API] Cannot find nvCVImage DLL or its dependencies ( code: NVCV_ERR_LIBRARY, op: NvAR_GetVersion )
LogMediaPipe: Error: Invalid state: MediaPipeContext is NULL.

Any ideas how to fix this?

我的demo，默认设置成 Mediapipe4u 的蓝图动画，按下P动作捕捉正常，但是按V切换了动画蓝图，再按P，就不能动捕了，那个OSC协议的再反复切换是正常的

MediaPipe4U Version

20230316

UE Version

5.0.x

UnrealEngine Type

Compiled From Source Code

What happened?

When I import a custom character in the demo project and follow the exact setup you did with the mannequin, the custom character works just fine. I tried to do it in a different project file and everything worked but within a few seconds of turning on the camera, the character seems to start flying in the air. I went back and forth and recreated all the steps but nothing seemed to solve the problem.

I have added a link to a folder containing two videos showing what I am talking about.

https://drive.google.com/drive/folders/1a14f-623EsApE0MHTzH2pnPDQ1G5W-U5?usp=share_link

再次感谢大佬，询问位移相关问题。
当设定如下，z改为1.8，标定倒计时0

1.输入一个4s视频，出现现象为：
（1）第一次播放，在原地摔倒，且摔倒在空中，hip保持不动，即无location位移

（2）后续的轮播：
人物上升，feet位置与原来hip平齐，摔倒动作正常(z轴有变化)

这个是因为蓝图中，mediapipe location solver我排在了最前面导致的问题。

2.标定，的原理是什么？标定地面吗？还是标注人物？还是建立坐标系？

MediaPipe4U Version

20230311

UE Version

5.0.x

What happened?

fsfsdfsdfsdf

First of all huge Thank you for your work! This whole project is amazing!

While testing the newest version of the patch (20230101) on the Metahuman skeleton inside of MediaPipe4U-Demo project I encountered a big glitch on right arm - it is very distorted and palm always tries to follow the left one. It only appears when "Use Arm IK" is activated. Same occurs with legs and Leg IK.

With "Use Arm IK" unchecked it works fine, but the elbow and shoulder distortions are then more pronounced of course.

Best Regards and Happy New Year!

运行期间点击弹出，有时候会卡住，然后发生崩溃

MediaPipe4U_Demo.log

MediaPipe4U Version

MediaPipe4U_UE5_0_230227

UE Version

5.0.x

UnrealEngine Type

Install From Epic Games Launcher

What happened?

[2023.03.15-07.46.30:155][465]LogWindows: Error: === Critical error: ===
[2023.03.15-07.46.30:155][465]LogWindows: Error:
[2023.03.15-07.46.30:155][465]LogWindows: Error: Fatal error!
[2023.03.15-07.46.30:155][465]LogWindows: Error:
[2023.03.15-07.46.30:155][465]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION 0x00007ff888f1fd38
[2023.03.15-07.46.30:155][465]LogWindows: Error:
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff888f1fd38 opencv_world341.dll!UnknownFunction []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a17dde7 IdolVerse.exe!FOpenCVWebcamProvider::Open() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\CameraProviders\OpenCVWebcamProvider.cpp:39]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a18d2ca IdolVerse.exe!UWebcamImageSourceComponent::Open() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\Core\ImageSource\WebcamImageSourceComponent.cpp:90]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a22a125 IdolVerse.exe!UMediaPipeHolisticComponent::StartInternal() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeHolisticComponent.cpp:137]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a229aa5 IdolVerse.exe!UMediaPipeHolisticComponent::StartImageSource() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeHolisticComponent.cpp:358]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a2298e3 IdolVerse.exe!UMediaPipeHolisticComponent::StartCamera() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeHolisticComponent.cpp:326]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65a259c9f IdolVerse.exe!UMediaPipeHolisticComponent::execStartCamera() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Intermediate\Build\Win64\UnrealGame\Inc\MediaPipe\MediaPipeHolisticComponent.gen.cpp:145]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b431f69 IdolVerse.exe!UFunction::Invoke() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b613052 IdolVerse.exe!UObject::CallFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6240df IdolVerse.exe!UObject::ProcessContextOpcode() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b64c865 IdolVerse.exe!UObject::execLetBool() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6272b1 IdolVerse.exe!ProcessLocalScriptFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b60340e IdolVerse.exe!ProcessScriptFunction<void (__cdecl*)(UObject * __ptr64,FFrame & __ptr64,void * __ptr64)>() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b626ba5 IdolVerse.exe!ProcessLocalFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6272b1 IdolVerse.exe!ProcessLocalScriptFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b60340e IdolVerse.exe!ProcessScriptFunction<void (__cdecl*)(UObject * __ptr64,FFrame & __ptr64,void * __ptr64)>() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b626ba5 IdolVerse.exe!ProcessLocalFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6240df IdolVerse.exe!UObject::ProcessContextOpcode() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6272b1 IdolVerse.exe!ProcessLocalScriptFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b60340e IdolVerse.exe!ProcessScriptFunction<void (__cdecl*)(UObject * __ptr64,FFrame & __ptr64,void * __ptr64)>() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b626ba5 IdolVerse.exe!ProcessLocalFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6272b1 IdolVerse.exe!ProcessLocalScriptFunction() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6264b1 IdolVerse.exe!UObject::ProcessInternal() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b431f69 IdolVerse.exe!UFunction::Invoke() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b6259c2 IdolVerse.exe!UObject::ProcessEvent() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff659188cd1 IdolVerse.exe!TMulticastScriptDelegate::ProcessMulticastDelegate() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65f22ab07 IdolVerse.exe!UButton::SlateHandleClicked() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65f1c56d6 IdolVerse.exe!TBaseUObjectMethodDelegateInstance<0,UButton,FReply __cdecl(void),FDefaultDelegateUserPolicy>::Execute() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65bbb8e0f IdolVerse.exe!SButton::ExecuteOnClick() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65bbe59f7 IdolVerse.exe!SButton::OnMouseButtonUp() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b9cc20d IdolVerse.exe!TSlotBaseSHorizontalBox::FSlot::operator []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b9c2276 IdolVerse.exe!TArray<TSharedPtr<SWindow,1>,TSizedDefaultAllocator<32> >::RemoveAll<<lambda_e2369253746a3fa7ae5830b136b55ef0> >() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65ba1945a IdolVerse.exe!FSlateApplication::RoutePointerUpEvent() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65ba0528a IdolVerse.exe!FSlateApplication::ProcessMouseButtonUpEvent() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b9fa998 IdolVerse.exe!FSlateApplication::OnMouseUp() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b78932a IdolVerse.exe!FWindowsApplication::ProcessDeferredMessage() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b777dc4 IdolVerse.exe!FWindowsApplication::DeferMessage() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b78b771 IdolVerse.exe!FWindowsApplication::ProcessMessage() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b7734a6 IdolVerse.exe!FWindowsApplication::AppWndProc() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff9a532e7e8 USER32.dll!UnknownFunction []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff9a532e229 USER32.dll!UnknownFunction []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65b78cb06 IdolVerse.exe!FWindowsPlatformApplicationMisc::PumpMessages() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff6590f5d95 IdolVerse.exe!FEngineLoop::Tick() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65910e77c IdolVerse.exe!GuardedMain() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff65910e84a IdolVerse.exe!GuardedMainWrapper() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff6591118cc IdolVerse.exe!LaunchWindowsStartup() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff659123084 IdolVerse.exe!WinMain() []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff661a286ba IdolVerse.exe!__scrt_common_main_seh() [D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff9a37b7614 KERNEL32.DLL!UnknownFunction []
[2023.03.15-07.46.30:155][465]LogWindows: Error: [Callstack] 0x00007ff9a56626a1 ntdll.dll!UnknownFunction []
[2023.03.15-07.46.30:155][465]LogWindows: Error:
[2023.03.15-07.46.30:158][465]LogExit: Executing StaticShutdownAfterError
[2023.03.15-07.46.30:160][465]LogWindows: FPlatformMisc::RequestExit(1)
[2023.03.15-07.46.30:160][465]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3)
[2023.03.15-07.46.30:160][465]LogCore: Engine exit requested (reason: Win RequestExit)
[2023.03.15-07.46.30:189][465]Log file closed, 03/15/23 15:46:30

以下是报错内容
Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:E:\Epic Games\UE_5.0\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 691] Array index out of bounds: 350 from an array of size 342

UnrealEditor_MediaPipe!UMediaPipeUtils::BuildBoneIndices() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeUtils.cpp:1048]
UnrealEditor_MediaPipe!UMediaPipeUtils::GetAllBonesInComponentSpace() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeUtils.cpp:1100]
UnrealEditor_MediaPipeBVH!AMediaPipeBVHRecorder::OnAddOn() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4UBVH\HostProject\Plugins\MediaPipe4UBVH\Source\MediaPipeBVH\Private\MediaPipeBVHRecorder.cpp:486]
UnrealEditor_MediaPipe!UMediaPipeAnimInstance::ReInitialize() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeAnimInstance.cpp:209]
UnrealEditor_MediaPipe!UMediaPipeAnimInstance::NativeBeginPlay() [F:\3D_Works\UE\MediaPipe4U\Build\MediaPipe4U\HostProject\Plugins\MediaPipe4U\Source\MediaPipe\Private\MediaPipeAnimInstance.cpp:258]
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

The crash is not happening, but also the video texture feed completely stopped, so is hand tracking feature.

你好，我下载了工程和插件，将他们整合在了一起，但是脸部的表情没有驱动，请问是什么原因。

大佬，春节快乐。
以下是我的问题。
先看截图吧







RemapAsset的骨骼映射我是按照您开发文档中的命名去选的，可能命名不一样，但骨骼选的应该是对的。
最后一张图就是开始摄像机捕捉时的画面，左侧自定义骨骼网格体只有手部和头部可以驱动，（我开了锁脊柱和腿，但不开也是一样效果），右侧Quinn是可以正常驱动的。
我自己的猜测，有可能是我的自定义骨骼有问题，看以下截图


以大臂为例，这个网格体的大臂骨骼是有两根可形变骨骼（具体怎么称呼我也不了解），我不知道这样的骨架构成会不会影响Mediapipe4U的自定义骨骼的动捕。我现在找到的大概有可能是这个问题，也有可能是其他问题，希望您能帮我解答一下。

十分感谢

运行程序的时候报 LogGStreamer: Error: GStreamer module is not available

执行到 蓝图Start函数时报 LogMediaPipeGStreamer: Error: GStreamer start fault !
filesrc location="C:/Users/xiaofan/Desktop/wiper.mp4" ! decodebin ! videoconvert ! video/x-raw,format=(string)RGBA ! videoconvert ! appsink name=mediapipe_sink

感谢作者的工作，我是纯新手，操作上可能会有问题，以下是我使用的步骤：
1.基于https://github.com/endink/MediaPipe4U-Demo
这个项目，使用VRM骨骼建立动画蓝图；

2. 修改character里面动画类和骨骼网络体；

3.运行后出现

4. 此时点击启用摄像头会崩溃

hi,
when I run mediapipe4u-demo,start image success,

but start video,no response.
Are there any restrictions on input video？

Hello,
Congratulations for this amazing plugin.
I'm testing your demo.
I want to know if it's possible to attach a video source (from a StreamMedia component or a MediaPlayer component) like a mp4 video or something else (without using virtual camera like OBS studio) instead of the webcam source.
I read the documentation and seems not be present something correlated.
Thank you.
Regards.

MediaPipe4U 版本

20220409

UnrealEngine 版本

5.1.x

UnrealEngine 类型

Launcher

问题类型

运行错误

错误详细描述

Generate project error
错误提示：Some Platforms were skipped due to invalid SDK setup: IOS, Android, Linux, LinuxArm64.
See the log file for detailed information

Discovering modules, targets and source code for project...
Visual Studio 2022 x64 must be installed in order to build this target.

但是已安装VS2022

日志文件：
UnrealVersionSelector-2023.04.13-16.03.40.log

Error message：
Missing precompiled manifest for 'MediaPipe', 'E:\MediaPipe4U-Demo\Intermediate\Build\Win64\MediaPipe4U_Demo\Shipping\MediaPipe\MediaPipe.precompiled'. This module was most likely not flagged for being included in a precompiled build - set 'PrecompileForTargets = PrecompileTargetsType.Any;' in MediaPipe.build.cs to override. If part of a plugin, also check if its 'Type' is correct.
UATHelper: 打包 (Windows):
I did every step of source code packaging, except”执行完成后将把链接符号拷贝道你的项目 Build 目录。“Can I ask what's going on？

UE版本：5.0.3
插件版本：20230106

在尝试根据视频教程使用自定义骨骼时发现Pose解算失效（身体部分不动）。Head，Hand以及Location可正常解算驱动

后尝试使用CharacterCreator模型再次根据视频使用自定义骨骼（不使用CharacterCreator预设），问题未得到解决

项目中使用的AnimationBluePrint及MediaPipeRemap资源如图

MediaPipe4U Version

2022.0.1

UE Version

5.0.x

UnrealEngine Type

Compiled From Source Code

What happened?

Hello everyone,
We have an rtmp stream from a remote pc and we are able to play this stream in a media player by attaching it to a stream media source.
We would to know if it is possible to give in input this stream media source to MediaPipe4U instead of the webcam or some video file.
We read the documentation and seems to be possible by blueprint only with a video file.
Thank you in advance.
Regards.

DEMO下载下来，按照说明编译，报错如下：
BlendShapeCalculator.h(11): [C1083] 无法打开包括文件: “MediaPipe/Private/Core/Filter/KalmanPoint.h”: No such file or directory
这个是什么问题？

When I am testing this in the UE editor with two players (multiplayer mode) , with Play as Client or Play as Listen Server option, the program crashes, and it gives an error at the MediaPipeHolisticComponent cpp class, which we do not have access to.
Also it doesn't seem to work with network replication at all, the pose solvers that we have set up in the animation blueprints, are moving all the players together in multiplayer mode, where actually it should just move the local player."

UE51试了一下无法编译demo，由于没有源码，看是否能编译一个ue51的版本，

我是在Take Recorder录制动画的时候出现的这个问题.

MediaPipe4U 版本

20230316

UnrealEngine 版本

5.1.x

UnrealEngine 类型

源码编译

问题类型

运行错误

错误详细描述

打开样例工程的NvAR_WithMediaPipe后，运行。

在开始游戏后的几秒内，pose动捕生效。然后会报错卡住。

报错如下：

Display LogMediaPipe [MediaPipe API] ------------> Start Loop Work Thread <------------
Display LogMediaPipe ImageSourceLoop Start! ThreadID = 43820
Display LogMediaPipe Webcam opened, capture resolution :1280x720 @ 30.0 fps
Display LogMediaPipe ImageSourceLoop thread enter.
Log Live Link (01) Can't evaluate frame for 'MediaPipe NvAR'. No data was available.
Error LogMediaPipeNvAR [NvAR API] The given parameter selector is not valid in this effect filter ( code: NVCV_ERR_SELECTOR, op: NvAR_SetU32(PoseMode))
Display LogMediaPipeNvAR NVAR Thread Start! ThreadID = 42780
Display LogMediaPipe [MediaPipe API] UmpPipeline::AddImageFrameIntoStream (in loop) OK.
Display LogMediaPipe [MediaPipe API] Notify image size received ( w: 1280, h: 720 ).
Display LogMediaPipe [MediaPipe API] First look workflow is done !
Display LogMediaPipe FDynamicTexture::Resize Width=1280 Height=720 Format=PF_B8G8R8A8
Display LogMediaPipe Bones for mediapipe joint cached.
Display LogMediaPipe Joints calibrated completed.

MediaPipe4U 版本

M4U_UE_5_0_20230409

UnrealEngine 版本

5.0.x

UnrealEngine 类型

Launcher

问题类型

项目打包

错误详细描述

尝试跟着文档构建新的动作捕捉项目并且打包时，在打包环节会碰到报错导致打包失败。
报错的内容是下边这些

后边尝试打包demo项目
同样也出现了这样的问题

MediaPipe4U 版本

20230316

UnrealEngine 版本

5.0.x

UnrealEngine 类型

Launcher

问题类型

运行错误

错误详细描述

设置如下：


效果如下（出现高亮部分并且离地更远）：

MediaPipe4U 版本

2023-04-03

UnrealEngine 版本

5.1.x

UnrealEngine 类型

Launcher

问题类型

运行错误

错误详细描述

Log.txt
自建Actor只包含MediaPipeHolistic，并在事件开始时Start Camera，拖入场景，运行后角色无法响应动捕。
使用Demo项目中的BP_MediaPipeRuntime拖入场景，角色可以正常响应

MediaPipe4U 版本

20230316

UnrealEngine 版本

5.0.3

UnrealEngine 类型

源码编译

问题类型

校准倒计时结束开始IK计算后的Foot骨骼位置一直在很上方的位置，导致人物在倒计时结束后会向上位移一段距离。

错误详细描述

设置如下：

地面的射线通道也设置了block
效果如下：（校准倒计时结束后人物就向上位移到了图片的位置）

射线检测的碰撞点确实是在地板上的，我看文档绿色盒子代表求解后的骨骼位置，效果是向上偏移了很多。

我看文档里面，GrouldSolveSettings分类里面应该是有这个属性的：FeetVerticalOffset，但是貌似并没有暴露出来，想着是不是可以通过这个参数调节一下呢？

期待收到您的回复！

MediaPipe4U 版本

20230409

UnrealEngine 版本

5.1.x

UnrealEngine 类型

Launcher

问题类型

项目打包

错误详细描述

大佬，又见面了，插件真的太棒了。
但我在打包时出现了一些问题，没有爆红，但最后打包失败了，我已经排除了插件以外的问题，无插件版本可以打包。
首先，我的系统是Win10，64位系统，
我按照您的更新文档中步骤，确认启用插件，LinkType也设置为了Default，创建了 copy_intermediate.bat并填好内容，并运行完毕，项目名称也改好了，但最后一部我不太理解，拷贝链接符号到Build目录这一步。
有两个问题，我的Build目录是指哪里，还有就是链接符号是指copy_intermediate.bat 这个文件吗？
还有个问题，源码版引擎是指在epic下载了源代码的引擎吗，不太了解。

希望您有时间可以给我个反馈。

Hello again Mr. Xiao! I didn’t find how to open issues on your MediaPipe4U-Demo Project, so I’ll post it here :)

I was testing out new update of Demo Project with the Ver. 20230104 of MediaPipe4U Plugin. With default Mannequin it works fine, but when I tried to test it on MetaHuman I encountered following problem:

If I want to replace Quinn with my Metahuman, Unreal crashes. When I want to delete Quinn from the scene I first get a warning that this will break references to MediaPipeDebugDrawer and BP_MediaPipeRuntime and after deleting Quinn and trying to Play the project UE crashes with following error:
Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000320

If I just try to add a Metahuman into existing scene without removing Quinn, UE crashes with a message:
Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:E:\Epic Games\UE_5.0\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 691] Array index out of bounds: 350 from an array of size 342

I created Animation Blueprint for MetaHuman following your tutorial on YouTube (UnrealEngine Mocap plugin --Customize skeleton) and it is just used on unchanged MetaHuman Blueprint as Animation Blueprint for the Body. With older versions of Demo it worked fine.

Am I missing something?

Best Regards,
Maria

首先感谢大佬！
如题，这块不知是插件的问题，还是说我这边配置的问题呢。
就是如果使用Custom骨骼绑定设置的话，人物就不动了，即使和您预设的UE5绑定成一样的骨骼也不行。
如果大佬能预定义一个Metahuman的骨骼就无敌了！

走在空气中的情形，脚步虚浮。
如何使其符合物理学，增加重力。
这是去年的一篇论文上，大佬有看过吗，有没有什么思路，还是说从ue这边加碰撞，和物理学重力之类的解决？
https://zhuanlan.zhihu.com/p/414125560

I have a problem when I choose a video to be played, nothing happened no video being displayed, I tried to convert the video using h.264 codec but It didn't change anything, I think It's related to "gstreamer". However images work fine, here is the log i copied in the unreal engine editor. Errors that is displayed in red is :

LogMediaPipeGStreamer: Error: GStreamer start fault !
filesrc location="D:/Videos/camjennie.mp4" typefind=true ! decodebin ! videoscale2 width=1280 height=720 ! videoconvert ! video/x-raw,format=(string)RGBA ! appsink name=mediapipe_sink

log.txt

First, Thanks for the Great Project you provided sir!
I have some Questions to ask.
Within all the Motion capture solution I know, Google mediapipe is the one I choose Because I want to capture both Face and Body Motion at the same time.
And I want to apply it to the Unreal Characters so, your project is spot on.
But I have a problem Applying OBS Virtual Camera as my input. (I use remote computing so, I send my image using https://vdo.ninja/ to remote computer and run the virtual cam using OBS)
The Unreal Log message tells me 'Unable to open camera 0' and,
I know camera 0 is Virtual OBS Camera beacuase I apply this method to 'Faceware' Software and it works perfectly.
wongfei's 'ue4-mediapipe-plugin' also have the same problem but at least I could try change the camera in the detail section in the pipeline option of the Deme Object in the example map. (Changing Camera also didn't works though in there)
Without OBS turns on, the error Message didn't pop up. But I prefer to use OBS Virtual Camera to use Remote computer.
So,

1. Is there any options I could find to change my webcam from webcam 0 to webcam 1 ?
2. What should I do to make your project recognise OBS virtual Camera?
3. Or Should I ask this to google-Mediapipe directly? Cause I presum it's problem that's more close to the Mediapipe itself, not yours.

The whole log Message is this.
LogMediaPipe: Display: [mediapipe_api.dll] UmpPipeline::Start
LogMediaPipe: Display: [mediapipe_api.dll] UmpPipeline::Start OK
LogMediaPipe: Display: [mediapipe_api.dll] Enter WorkerThread
LogMediaPipe: Display: [mediapipe_api.dll] UmpPipeline::Run
LogMediaPipe: Warning: [mediapipe_api.dll] FaceGeometry is enabled, auto disable refine_face_landmarks options.
LogMediaPipe: Display: [mediapipe_api.dll] CalculatorGraph::Initialize
LogMediaPipe: Display: [mediapipe_api.dll] VideoCapture::open
LogMediaPipe: Error: [mediapipe_api.dll] Unable to open camera 0
LogMediaPipe: Error: [mediapipe_api.dll] Unable to open camera 0
LogMediaPipe: Display: [mediapipe_api.dll] UmpPipeline::Stop
LogMediaPipe: Error: [mediapipe_api.dll] resource deadlock would occur: resource deadlock would occur
LogMediaPipe: Display: [mediapipe_api.dll] CalculatorGraph::CloseInputStream: 0
LogMediaPipe: Error: [mediapipe_api.dll] bad function call
LogMediaPipe: Display: [mediapipe_api.dll] Leave WorkerThread

Thanks

Version 5.0 has an exe file in the Binary folder which launch the plugin with the character of my choosing. The problem is by default it selects Device Id 0 but I use a camera with DeviceID 3. I tried to change in BP_Mediapiperuntime but nothing works in the exe. In the editor, it detects the change. Is there any way I could include the option to change the device id after running the exe and from the window just like other options like record bvh, pose correction, head movement etc? I played with the BP to integrate such a thing but no luck. A little bit of direction will be helpful :)

MediaPipe4U Version

20230316

UE Version

5.0.x

UnrealEngine Type

Install From Epic Games Launcher

What happened?

Warning LogGStreamer GstPipeline <MediaPipe_GS_Pipline> BUS WARNING: NO_ERROR_MESSAGE
Error LogGStreamer GstPipeline <MediaPipe_GS_Pipline> BUS ERROR: Internal data stream error.

问题：我用我的动画蓝图重设父类为MediaPipeAnimInstance，但输出日志还是打印：LogMediaPipe:错误:PoseSolver为null点，FAnimNode_MediaPipePose只能在MediaPipeAnimInstance中使用。并且报警告：Unable to initialize bones to modify, hands solver was not initialized。

我的BonePreset用的自定义骨骼，也在Bone Setting匹配好了骨骼链，然后日志就报错。

我不知道这个问题属不属于您的解决范围，如果不是，您就忽略我的问题就好了。
我尝试使用一个UE5推流OBS的方案，使用了插件：OWLLivestreamingToolkit，
然后在Rider重新编译后，运行UE5出现了报错，Rider编译没有错误，运行时才会报错，见截图：

通过排除法，我发现在Mediapipe和那个插件同时存在时，就会出现上边的错误，我在那边也反馈了错误，希望可以得到解决。

谢谢您

Gobcurry

hi , i was unable to build the project at visual studio , i'm so sorry since i'm new to this field , i cant even find a suitable tutorial to export log activity from visual studio , all the tutorial are using the tools which i dont have in my computer
嗨，我无法在 visual studio 上build Ue5和mediapipe4u，我很抱歉，因为我是电脑白痴，我找不到合适的教程来从 visual studio 导出日志活动，我的电脑里没有教程所使用的工具

Output-Build.txt

My Log is shown as follows. Could you help me find out the problems? Thanks!
(FYI, I set up every preparations as your doc) : )
LogMediaPipe: Display: Right Thumb Axis: X=-0.000 Y=-0.000 Z=-1.000
LogMediaPipe: Display: Joints calibrated (RightHandSolver) .
LogMediaPipe: Display: Bone mapped: LeftShoulder ----> [11]upperarm_l ( parent: [10]clavicle_l )
Bone mapped: RightShoulder ----> [57]upperarm_r ( parent: [56]clavicle_r )
Bone mapped: LeftElbow ----> [12]lowerarm_l ( parent: [11]upperarm_l )
Bone mapped: RightElbow ----> [58]lowerarm_r ( parent: [57]upperarm_r )
Bone mapped: LeftWrist ----> [20]hand_l ( parent: [12]lowerarm_l )
Bone mapped: RightWrist ----> [66]hand_r ( parent: [58]lowerarm_r )
Bone mapped: LeftPinky ----> [26]pinky_01_l ( parent: [25]pinky_metacarpal_l )
Bone mapped: RightPinky ----> [72]pinky_01_r ( parent: [71]pinky_metacarpal_r )
Bone mapped: LeftIndex ----> [37]index_01_l ( parent: [36]index_metacarpal_l )
Bone mapped: RightIndex ----> [83]index_01_r ( parent: [82]index_metacarpal_r )
Bone mapped: LeftThumb ----> [34]thumb_02_l ( parent: [33]thumb_01_l )
Bone mapped: RightThumb ----> [80]thumb_02_r ( parent: [79]thumb_01_r )
Bone mapped: LeftHip ----> [129]thigh_l ( parent: [1]pelvis )
Bone mapped: RightHip ----> [106]thigh_r ( parent: [1]pelvis )
Bone mapped: LeftKnee ----> [130]calf_l ( parent: [129]thigh_l )
Bone mapped: RightKnee ----> [107]calf_r ( parent: [106]thigh_r )
Bone mapped: LeftAnkle ----> [131]foot_l ( parent: [130]calf_l )
Bone mapped: RightAnkle ----> [108]foot_r ( parent: [107]calf_r )
Bone mapped: LeftHeel ----> [131]foot_l ( parent: [130]calf_l )
Bone mapped: RightHeel ----> [108]foot_r ( parent: [107]calf_r )
Bone mapped: LeftFootIndex ----> [132]ball_l ( parent: [131]foot_l )
Bone mapped: RightFootIndex ----> [109]ball_r ( parent: [108]foot_r )
Bone mapped: Hip ----> [1]pelvis ( parent: [0]root )
Bone mapped: Head ----> [9]head ( parent: [8]neck_02 )
Bone mapped: Neck ----> [7]neck_01 ( parent: [6]spine_05 )
Bone mapped: Spine ----> [2]spine_01 ( parent: [1]pelvis )
LogMediaPipe: Display: LocationSolver: 38 bone cached
LogGStreamer: Error: GstPipeline: BUS ERROR <MediaPipe_GS_Pipline>
LogMediaPipe: Display: [mediapipe_api.dll] CalculatorGraph::StartRun
enable_segmentation : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
input_horizontally_flipped : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
input_rotation : mediapipe::Packet with timestamp: Timestamp::Unset() and type: int
input_vertically_flipped : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
model_complexity : mediapipe::Packet with timestamp: Timestamp::Unset() and type: int
refine_face_landmarks : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
smooth_landmarks : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
smooth_segmentation : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
use_prev_landmarks : mediapipe::Packet with timestamp: Timestamp::Unset() and type: bool
LogMediaPipe: Display: [mediapipe_api.dll] ------------> Start Loop Work Thread <------------

MediaPipe4U 版本

Mediapipe4u-Demo(2023/4/3)

UnrealEngine 版本

5.1.x

UnrealEngine 类型

Launcher

问题类型

其他

错误详细描述

可以动捕，但是捕捉的很奇怪，求大佬帮忙，我是计算机小白，只会用大佬的软件

MediaPipe4U Version

20230316

UE Version

5.0.x

UnrealEngine Type

Install From Epic Games Launcher

What happened?

Can't run with video!
MediaPipeHolisticComponent, and set SourceResolutionLimits to None

Video test:

When I try to package an exe with my custom model or even the included model, after a while I get the error below. Could you help me out with it. I have seen the error in 5.0 as well.

### Packaging (Windows): Missing precompiled manifest for 'MediaPipe', 'D:\MoCap\MediaPipe4U-Demo-main-March2023\MediaPipe4U-Demo-main\Plugins\MediaPipe4U\Intermediate\Build\Win64\UnrealGame\Development\MediaPipe\MediaPipe.precompiled'. This module was most likely not flagged for being included in a precompiled build - set 'PrecompileForTargets = PrecompileTargetsType.Any;' in MediaPipe.build.cs to override. If part of a plugin, also check if its 'Type' is correct.
UATHelper: Packaging (Windows): Took 12.569327600000001s to run dotnet.exe, ExitCode=6
UATHelper: Packaging (Windows): UnrealBuildTool failed. See log for more details. (C:\Users\Opu\AppData\Roaming\Unreal Engine\AutomationTool\Logs\D+Games+UE_5.1\UBT-MediaPipe4U_Demo-Win64-Development.txt)
UATHelper: Packaging (Windows): AutomationTool executed for 0h 0m 17s
UATHelper: Packaging (Windows): AutomationTool exiting with ExitCode=6 (6)
UATHelper: Packaging (Windows): BUILD FAILED
PackagingResults: Error: Unknown Error

使用继承MediaPipeInstance 如下：

在运行时修改动画蓝图或设置为空，结束运行时报错，报错如下

MediaPipe4U 版本

20230309

UnrealEngine 版本

5.0.x

UnrealEngine 类型

Launcher

问题类型

插件安装

错误详细描述

关闭ue引擎，下载新的license文件并覆盖原文件后，重新打开并运行，动补失效（点击start web camera没有相应），查看许可证仍然显示3.15到期。请教一下是哪里出了问题。

插件有效期是多久，如果到期如何获得授权呀