Comments (32)
@chrisywong
Sounds like your images are published in the frame of camera_color_optical_frame
but the transforms are being done with respect to camera_depth_optical_frame
. The 0.02m shift makes sense as the depth lens is more or less 2cm away from the color lens on a realsense. Since you are already using /camera/aligned_depth_to_color/image_raw
I would change the frame to camera_color_optical_frame
in the location that Ravi linked above, which should solve the problem.
from ros_openpose.
Thanks for the reply @ravijo
I'm surprised message_filters::sync_policies::ApproximateTime
didn't result in good performance. In my experience, messages are received at a stable rate so long as the image and depth stream have a similar fps.
I will try to use opWrapper.waitAndEmplace()
and see if there is a noticeable boost in performance.
from ros_openpose.
Hi, frankly speaking, while writing this package, my application was not expecting large movements and such nice precision. I did not verify this 2cm shift.
However, what I have seen that there were some similar-looking TF frames for the camera. Right now, I am using camera_depth_optical_frame
as shown here.
You can echo all TF frames for the camera and there is a possibility that you can find similar TF (no relative rotation and no shift in the z-direction). If so, please change this parameter. I am not so sure though.
The other hack can be publishing a child TF of camera_depth_optical_frame
with no rotation but only 2cm shift in x (or y) direction as you noticed.
from ros_openpose.
I understand.
I think it is all right to submit a pull request for the Python version only. I will create a new branch, say python
, and will be adding to it.
from ros_openpose.
I apologize for the delay. I am away from my lab for a few days and I want to fully bug-proof the code on a realsense camera before submitting a pull request. I should be back in my lab and be able to get my hands on the camera come this Thursday and will submit it then.
from ros_openpose.
@ravijo @chrisywong
Just an FYI that the pull request is submitted. Sorry for the delay, got access to a Realsense just today.
From tests, latency is greatly reduced on synchronous version but has worse frame rate than the original wrapper so the best version to use will depend on the usage case.
from ros_openpose.
Thank you so much @QuantuMope
I sincerely acknowledge your contribution.
The latest commit 0ed2a72 now includes support for the synchronous version provided by you.
In the documentation, I quickly added the relevant information under Implementation Versions Info..
Thank you again.
You may like to pull the latest changes from the repository. Please let us know if you face any issues.
from ros_openpose.
Sorry to hear about the issue.
Thank you @luorui93 for providing detailed explanations to fix the problem. You have clearly stated all possible ideas! Thanks again.
In the case of the ImportError
exception, I always try to import the module manually in python interpreter first to make debugging a lot easier.
First, please switch to tutorial_api_python
folder provided by OpenPose team and then import pyopenpose
as shown below:
ravi@dell:~$ cd /home/ravi/openpose/examples/tutorial_api_python
ravi@dell:~$ python
Python 2.7.17 (default, Feb 27 2021, 15:10:58)
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('../../python')
>>> from openpose import pyopenpose as op
>>> exit()
Above, I assumed that you are going to use pyopenpose
module with standard ROS. By saying standard ROS, I just mean that ROS which by default works with Python 2 version. However, you are in Ubuntu 20, you must be using ROS with Python3. So in such case, please use python3
to import pyopenpose
as shown below:
ravi@dell:~/openpose/examples/tutorial_api_python$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('../../python')
>>> from openpose import pyopenpose as op
>>> exit()
To see all python versions available, please type python
and then press the tab key twice. You may see similar results:
ravi@dell:~$ python
python python2-pasteurize python3.6m-config python3m-config
python2 python2-qr python3.7 python-config
python2.7 python3 python3.7m pythontex
python2.7-config python3.6 python3.8 pythontex3
python2-config python3.6-config python3-config
python2-futurize python3.6m python3m
If you can not import pyopenpose
as shown above, I highly doubt that the issue is related to OpenPose installation/building. Please follow the installation documentation properly.
Next, please read the following:
- Please identify the
python
version of your ROS. You must use the same python for building OpenPose (as said by @luorui93 too). To find out your ROS python version,PYTHON_EXECUTABLE
,PYTHON_INCLUDE_DIR
, andPYTHON_LIBRARY
variables, you can write the following code in any of your working ROS packages:Then please execute the script asimport rospy import sys from distutils.sysconfig import get_python_inc import distutils.sysconfig as sysconfig if __name__ == '__main__': rospy.init_node('example_node', anonymous=True) print(sys.version) # may print as '2.7.17 (default, Feb 27 2021, 15:10:58) \n[GCC 7.5.0]' print(sys.executable) print(get_python_inc()) print(sysconfig.get_config_var('LIBDIR'))
rosrun my_package my_script.py
- Last but not least, I also noticed that you have kept OpenPose inside the ROS workspace. Inside a ROS workspace, we are required to execute
source devel/setup.bash
. This changes many environment variables. I suggest you keep OpenPose in$HOME
or somewhere else so that OpenPose installation doesn't get affected by ROS workspace.
PS: I have reopened this issue. However, I kindly request you to open a new issue with these details. Because this issue, i.e., #31 was originally created for discussion on processing delay.
from ros_openpose.
Solved this issue, see #41.
from ros_openpose.
I was just about to post about this problem too. I'm thinking the delay is from openpose
itself (since I notice the lag there). Maybe the real time output from the example gif is using saved data rather than an actual real time feed?
This delay also produces another issue particular to the 3D projection and fast motions. If the motion is fast enough (e.g. swinging your arms quickly) that the delayed skeleton no longer overlaps with the person, the skeleton actually gets projected onto the background. What this tells me is that perhaps the data used for 3D projection in ros_openpose
is using unsynced (or improperly synced) keypoints from openpose
and the real time point cloud data.
E.g. in the screenshot below where I'm waving my arm, OpenPose is displaying an old image, which corresponds to the position of the skeleton, but the point cloud image is more updated and showing my arm already extended. You can't see as well but the skeleton keypoint is projected onto the background.
I'm trying to see if I can do anything with timestamps to try and make it work.
from ros_openpose.
Yes, this is pretty much exactly the problem I am having as well. If arm movements are too fast, the skeleton lags behind considerably as you've shown.
I was unaware that the 3D projection actually uses the current frame. That would indeed result in quite inaccurate results.
I can try to see if this is also the case on my end and report back.
If the delay is indeed solely due to openpose, it seems like there's no clear solution other than to find other more efficient methods for skeleton recognition. This is my first time using openpose but I was led to believe that it was more-or-less suitable for realtime operation.
from ros_openpose.
From what I see, the delay isn't TOO bad (<1 sec). If we can somehow figure out the exact delay from OpenPose, we could feed the old colour/depth images and at least have them sync'd up.
Also I'm not 100% sure what I said is the case. This is just my deduction so far from digging into the code, but I'm not deeply familiar with it just yet.
from ros_openpose.
From openpose forum posts I've read so far, it seems like we should expect a 250 ms or less delay with a good enough GPU. I think the delay exceeds this though.
I did notice that in the code the ros header time is assigned by the output worker after the openpose computation. This would result in skeleton frames being published with a time stamp that is later than the actual input time which results in the delay/mismatch.
from ros_openpose.
Hello @QuantuMope and @chrisywong
Thank you very much for bringing up this issue.
I was using GTX 1080 with Intel i7 having a 16GB memory machine. My environment details can be found here
I don't remember but back then I compiled many packages (even librealsense
too, maybe OpenCV with Cuda support too (unsure about OpenCV, I forgot)) from the source. In one of my projects, I have noticed the compiling packages from source gives better performance. So, please give it a try in this way if you get time.
Next, the 3D projection code is not optimized right now. You can see that for each detection keypoint in 2D space, the 3D point is calculated in a sequence. I did use OpenMP but I think that the overhead of creating threads is too much in our case. However, in principle, all 2D points can be converted to 3D using parallel computation to speed up the delay.
Another thing you assumed correctly is about the data used for 3D projection in ros_openpose
. Right now, the latest available depth image is used for 3D projection. Please see here. In the header file, this is what happening. While developing this package, as two independent workers were created (op::WorkerProducer
and op::WorkerConsumer
), I assumed that it is more or less will work properly. I remember that in order to use the depth image corresponding to the color image, once I locked the depth image while acquiring the color image and later on used that depth image after receiving key points from OpenPose, it worked. I wasn't satisfied with the overall performance.
In the past, you may notice in older commits, that I used message_filters::TimeSynchronizer
from ROS but it actually added delay because it needs to wait until it receives frames with the same timestamp. Please see here. In this implementation, we need to provide queue size to the TimeSynchronizer
. At that time, I wanted to make the queue size as small as possible but it turned out that this number somehow depends on the processing capabilities of the machine. I do remember that I tried message_filters::sync_policies::ApproximateTime
given by ROS too but I wasn't satisfied by the overall performance.
Other than this, once I remember one person said that opWrapper.waitAndEmplace()
style is giving better performance. You can see an example code here. I have not checked it but you can easily code it in ros_openpose
with minimal effort. Please give it a try if necessary.
Thank you so much. Please let me know the outcome.
from ros_openpose.
Have either of you noticed that the skeleton and point cloud are not aligned in RViz? All points seem to be shifted to the left by about 0.02m. I just want to see if it's a general thing or if it's something related to my setup and/or parameters. I'm using /camera/aligned_depth_to_color/image_raw
for my Intel RealSense D435.
from ros_openpose.
@QuantuMope Perfect 👍
This is what I was trying to say. It should work. Thanks a lot.
from ros_openpose.
Hello @chrisywong and @ravijo.
I apologize for the late update.
Just wanted to update that I created a new ros wrapper using message_filters::sync_policies::ApproximateTime
and opWrapper.waitAndEmplace()
and was able to reduce latency considerably. On my system, I am experiencing an average of 100 ms per frame. I think if real-time performance is important, it is worth going with a synchronous approach. I will close this issue since the problem has at least been solved for me.
from ros_openpose.
Hi @QuantuMope
Thank you so much for the update. I am glad that the approach worked well on your side.
By the way, would you mind making a pull request to share your code?
I am thinking to add your contribution to the dev
branch by acknowledging your kind efforts too.
from ros_openpose.
Sure but I just wanted to clarify two things.
- The code I wrote does not reuse any of the original
ros_openpose
code. Since your code is asynchronous, I thought it would be faster to just rewrite the code from scratch. Still it could prove useful to have two separate modules. One that prioritizes latency (synchronous) and one that prioritizes fps (asynchronous). - I wrote the ros wrapper in Python for speedy prototyping.
Given this, let me know if you'd still like me to submit a pull request. If you'd like the code to stay C++ (since using openpose through Python requires additional compilation steps), if I have some time, I can even convert the code to C++ and submit a pull request then. Just let me know.
from ros_openpose.
Hi @QuantuMope, I'd definitely be interested in trying out your synchronized version! I'm not 100% sure which one I'll end up using for my application, but it'd be good to compare to see which one is more suitable for me.
from ros_openpose.
If you are in hurry, implementing opWrapper.waitAndEmplace()
and message_filters::sync_policies::ApproximateTime
is rather easy once you already have ros_openpose
. I did that in the past but did not push it to GitHub.
Meanwhile, we can wait for @QuantuMope for a pull request.
from ros_openpose.
Have a nice relaxing break @QuantuMope
from ros_openpose.
Hello @ravijo,
Just wanted to notify that I plan on implementing the synchronous node in C++ when I have some free time in the coming weeks. Will submit a pull request when I have it complete and tested. It may be worth keeping both the C++ and Python versions though in case users want to modify the nodes in their language of choice.
from ros_openpose.
Hello @QuantuMope
Thank you so much.
Yes, I agree with your suggestion.
Thanks again.
from ros_openpose.
Hi all,
I'm trying to use the synchronous script but am running into issues. When I try to run the launch file, I get this error:
[ERROR] [1617024825.486954]: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?
ImportError: cannot import name pyopenpose
I did enable BUILD_PYTHON
when building OpenPose (both v1.7.0 and v1.6.0), as:
$ find / -iname "pyopenpose*"
/usr/local/python/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/CMakeFiles/pyopenpose.dir
This occurs on the line from openpose import pyopenpose as op
in ros_openpose_synchronous.py
with the default path <arg name="py_openpose_path" default="/usr/local/python/" />
Any ideas?
Would this somehow be a conflict between having both Python 2.7 and 3.6 on my system?
from ros_openpose.
Hi all,
I'm trying to use the synchronous script but am running into issues. When I try to run the launch file, I get this error:
[ERROR] [1617024825.486954]: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder? ImportError: cannot import name pyopenpose
I did enable
BUILD_PYTHON
when building OpenPose (both v1.7.0 and v1.6.0), as:$ find / -iname "pyopenpose*" /usr/local/python/pyopenpose.cpython-36m-x86_64-linux-gnu.so /usr/local/python/pyopenpose.cpython-37m-x86_64-linux-gnu.so /usr/local/python/openpose/pyopenpose.cpython-36m-x86_64-linux-gnu.so /usr/local/python/openpose/pyopenpose.cpython-37m-x86_64-linux-gnu.so /usr/local/python/openpose/CMakeFiles/pyopenpose.dir
This occurs on the line
from openpose import pyopenpose as op
inros_openpose_synchronous.py
with the default path<arg name="py_openpose_path" default="/usr/local/python/" />
Any ideas?
Would this somehow be a conflict between having both Python 2.7 and 3.6 on my system?
First make sure you've successfully built the python openpose. Check the if there is a python folder in your openpose build folder.
If you've already built it without problems, to use the default "/usr/local/python" as the openpose module path you have to do
sudo make install
in the build folder to install the modules.
Or you can just change the default path in launch file to something like "path_to_your_openpose/build/python"
And if your default python for ROS is python2, you have to make sure you built openpose with python2 too. The default one is python3, you have to change the cmake flag in configuration.
Refer to this link:
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/03_python_api.md#compatibility
from ros_openpose.
I did do sudo make install
(that's why I have those files located in /usr/local/python/
). Pointing directly to ~/catkin_ws/src/openpose/build/python/openpose/
did not work either.
from ros_openpose.
I did do
sudo make install
(that's why I have those files located in/usr/local/python/
). Pointing directly to~/catkin_ws/src/openpose/build/python/openpose/
did not work either.
Then it's probably you built a python 3 module but your ROS is looking for a python 2 version. Check the link I pasted before and make sure you build with Python2 executable and libraries.
from ros_openpose.
Tried building Python 2 vs 3 and still nope. Even pointing to build folder specifically for python 2.7 (and verified version in CMakeCache.txt)
from ros_openpose.
I did do
sudo make install
(that's why I have those files located in/usr/local/python/
). Pointing directly to~/catkin_ws/src/openpose/build/python/openpose/
did not work either.
I think you should point to absolute_path/catkin_ws/src/openpose/build/python
. Also don't use "~", type the absolute path like /home/user_name/catkin_ws/.....
. Python won't interpret "~"
And maybe you can double check if you have a similar layout like this in your build folder if you run $ ls python/openpose
$ ls openpose/
CMakeFiles __init__.py Makefile
cmake_install.cmake __init__.pyc pyopenpose.so
from ros_openpose.
Yes, I am using the absolute path. I just wrote ~
here for convenience.
Neither py2.7 nor py3.6 build/python/openpose/
folders has __init__.pyc
, while py2.7 has pyopenpose.so
but py3.6 has pyopenpose.cpython-36m-x86_64-linux-gnu.so
(which, if I rename/copy/symlink to pyopenpose.so
still doesn't help)
from ros_openpose.
pyc file is not important, as long as you have __init__.py
.
Looks like your module should be built correctly.
By the way, my last comment not only points out the absolute path, you should also only set the path to .../build/python
instead of .../build/python/openpose
.
If that's still correct, I'm out of ideas...
from ros_openpose.
Related Issues (20)
- Increasing frame rate for Azure Kinect HOT 3
- Cannot get it to work in the most standard configuration HOT 2
- Frame_id + Publishing depth information HOT 6
- Information about Face Output HOT 7
- Hand skeleton error HOT 4
- Net Resolution HOT 2
- Only Hand Poses HOT 1
- Skeleton error in Rviz HOT 2
- Directory for openpose python node HOT 6
- encoding MONO16 support HOT 5
- Cmake Error - Package Configuration File Not Found HOT 2
- Error of run.launch HOT 14
- Input the Wrapper a recorded Azure Kinect Video HOT 2
- (OpenPose 1.7.0:16877): Gtk-ERROR ** HOT 5
- Need to detect only one person in the frame HOT 2
- Missing tags section HOT 2
- REQUIRED process [rosOpenpose-2] has died! HOT 4
- Lunch file operation error HOT 6
- Error running the launch file run.launch HOT 2
- Question about ROS Message Frame 3D Points HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ros_openpose.