Video black, hand capture not work about mediapipe4u-plugin HOT 15 CLOSED

Which version worked well with the video tracking on your PC?

from mediapipe4u-plugin.

I mean the version of MediaPipe4U

from mediapipe4u-plugin.

I don't think third-person mode is the key, it's just the difference between the perspective and the controller. So, I just used Character as a pawn, and use network mode, set player num to 2, it worked well on my PC. I've push this test case to github (demo project) .

There is a new level called PawnMode in the Maps folder, all you have to do is set the number of players and adjust the server mode, this Level override the GameMode, and UsePawn = true (new variable in BP_MediaPipeRuntime).

from mediapipe4u-plugin.

The process is documented here：

from mediapipe4u-plugin.

Can you show me that it is working for two players, I mean if you wave the hand then it is replicated in the server side properly. To display this , follow the below steps :

1. Add two playerstart in the scene, so two players will be joining.
2. Make them face each other, since you have not added any movement input , the camera movement wont work for your case. So it is better to make the two playerstart face each other so they both will be visible in the default camera view
3. Now in the first window try to wave the hand
4. In the second window see if the other player is waving the hand properly.
5. Please keep in mind that when you wave the hand in a specific window, only that player will wave the hand in that window and not the other player, while in the other window, the player standing at the opposite end will wave the hand.

Expected outcomes : 1 on waving hand, both players should not wave hand simultaneously, only the respective player character
2. for the other window, the player standing at the opposite side should wave the hand.

Current outcome : 1. when we wave the hand, both the players are waving the hands , and not just the client player
2. When the other player waves his hand, I cannot see that change in my machine.

from mediapipe4u-plugin.

You need to manually control whether the Animation Blueprint connects to mediapipe, now it is automatically connected, in fact, you can also connect yourself, but this lacks documentation, simply say:

1. Set the AutoConnectToMediaPipe property of the MediaPipeAnimInstance (your anim blueprint) to false
2. Call "MediaPipeAnimInstance::ConnectToMediaPipeInLevel" int begin play event or when you need it.
3. [Optional] Call "MediaPipeAnimInstance::DisconnectFromMediaPipe" when you don't need it anymore.

In your case :

1. Listener server is not a server only, it is also a client, only the dedicated server is. To determine server and client, you need to implement the logic yourself, it has many ways, for example, check Authority, chek Pawn, check Player, and so on.

2. About "When the other player waves his hand, I cannot see that change in my machine."
   I think you need to be clear about some basic things .In the design of Unreal Engine, most of components like the Character's Skeletal Mesh and its Animation Blueprint are not replicated, only variables that like a Character's velocity, postion are replicated, you want to replicate animation, you need to go deep into the animation instance or animation node to write code.

A simple way is read the skeleton transform put theme into variables , and replicate theme through network.

Whether 1 or 2, this is the topic of UE multiplayer development, and has nothing to do with this plugin.

from mediapipe4u-plugin.

The rest of the animation like walking running and jumping are replicated perfectly in animation blueprint. Only the part where I am calling your mediapose solver is not replicated

from mediapipe4u-plugin.

Those animations are just copies of some external variables (numbers), it copies these variables and then executes them again with the same animation Blueprint, the external variables for mediapipe animation are video streams, which show that it cannot be simply copied, so if you want to replica the mocap animation, the plugin does not support it, you need to develop it yourself, read the bone data (Pose) as a variable and replica them.

from mediapipe4u-plugin.

Do we need the source code for that ? Because I dont understand how can I read your bone data from these nodes

from mediapipe4u-plugin.

Reading skeletal data is a development thing, it's not that my plugin organizes you to read it, it's always there, it just needs to be programmed，sorry, my plugin doesn't have a feature capable of doing this

from mediapipe4u-plugin.

So you mean , I can write the code in C++ myself , by extending your base classes ? Even if I dont have access to the source ? Or I need to get your code so that I can modify it ?

from mediapipe4u-plugin.

Unreal engine already provides many ways to read skeletal data, such as writing a custom animation blueprint node, such as doing it in tick, this does not require extension plugin, plugin and this matter are independent

from mediapipe4u-plugin.

That I already know, but since this animbp class has been created by you in C++ , so these nodes also need to be replicated first in C++ so that we can read the data accordingly. So without extending your animbp class , how is that possible ? It is not just we are reading any random data but your pose solver data , which needs to be accessed in C++.

from mediapipe4u-plugin.

Let me briefly say that the plugin does not have the feature of playing bone data, and does not any feature to help you to read bone data or replicate bone data.

1. About "replicate skeletal data", I'm just providing a simple idea. Not necessarily the answer to the question and best practice, your question is how to replicate animations. This plugin can't help you do these things, maybe there are other plugins that can.

2. After replicate the bone data, how to play these bone data again also needs to be developed by yourself. It could be another Animation Blueprint ( no mediapipe4U ), and these are things you developed yourself.

If something you are working on is blocked by this plugin, you can post an new issue, if the plugin has worked for mocap now, I will close this issue.

from mediapipe4u-plugin.

The document of the Unreal engine show a samle: "Replicating the Player's Health with RepNotifies"，hope you find ideas:

https://docs.unrealengine.com/4.27/en-US/InteractiveExperiences/Networking/QuickStart/

I close issue now. good luck !

from mediapipe4u-plugin.