If we want to just dump the JSON estimation information on a video, and not look at it (or generate images from it), the video loops after completion. It makes sense to probably check out the results on the fly and loop, but from a pre-processing or meta-data extraction point of view, maybe better to include a flag to stop looping?

This is currently ~line 540 of rtpose.cpp

Are there any options to increase fps besides reducing resolution or adding GPUs? Is it possible to restrict detection to certain joints (e.g. Heads) in order to speed up processing?

finally get everything to compile on Ubuntu 16.04/opencv3.2.0/cuda8/cudnn6, but got the following at the first run:

F0415 22:05:23.604416  3325 common.cpp:151] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
*** Check failure stack trace: ***
    @     0x7fd156ebf2d2  google::LogMessage::Fail()
    @     0x7fd156ebf216  google::LogMessage::SendToLog()
    @     0x7fd156ebebe8  google::LogMessage::Flush()
    @     0x7fd156ec1f63  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fd157764782  caffe::Caffe::SetDevice()
    @           0x40a276  warmup()
    @           0x411184  processFrame()
    @     0x7fd152af36ba  start_thread
    @     0x7fd15282982d  (unknown)
Aborted (core dumped)

Please help

Recently, tensorflow 1.3 has updated to cudnn 6, so I updated cudnn to 6.

As I exec./install_caffe_and_cpm.sh, there are compiling error with ./include/caffe/util/cudnn.hpp.

For a few time trying, I find the newest caffe has different version cudnn.hpp.

Then I replace this hpp file with the newest one, it works fine.

Encountered compilation error while building rtpose.cpp.
Any help is highly appreciated.

examples/rtpose/rtpose.cpp: In function ‘void* processFrame(void*)’: examples/rtpose/rtpose.cpp:1185:42: error: no match for ‘operator[]’ (operand types are ‘boost::shared_ptr<float []>’ and ‘int’) frame_batch[n].joints[ij] = joints[ij];

gcc (Ubuntu 4.9.4-2ubuntu1~12.04) 4.9.4
boost 1_49_0

I have compiled rtpose-caffe with cuDNN. (I have set USE_CUDNN:=1)
but when I test in just 2 image ,it show "out of memory"

[lfx@gpu-com image]$ ls
image2.jpg image3.jpg

[lfx@gpu-com caffe_rtpose-master]$ ./build/examples/rtpose/rtpose.bin --image_dir image/
F0407 10:27:17.102958 1129 syncedmem.cpp:64] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7efeaa736e6d (unknown)
@ 0x7efeaa738ced (unknown)
@ 0x7efeaa736a5c (unknown)
@ 0x7efeaa73963e (unknown)
@ 0x7efeb1905912 caffe::SyncedMemory::to_gpu()
@ 0x7efeb1904c79 caffe::SyncedMemory::gpu_data()
@ 0x7efeb177e762 caffe::Blob<>::gpu_data()
@ 0x7efeb197b9d3 caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7efeb18cdb8b caffe::Net<>::ForwardFromTo()
@ 0x40bcff warmup()
@ 0x411cbb processFrame()
@ 0x7efe9ed99dc5 start_thread
@ 0x7efe9eac873d __clone
Aborted (core dumped)

I have two GPUs(GT730+TITAN X), it seems already run in TITAN X, but i can't make sure

how can I make sure it run in TITAN X?
If it already run in TITAN X, how can i fix the error: out of memory?

I got one board Jetson TX2. I am trying to compile and run rtpose to see how it performs on Jetson TX2.
I compiled with no problem. However, When I run it. I got the following error:
F0607 11:47:28.814931 13654 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f9a445718 google::LogMessage::Fail()
@ 0x7f9a447614 google::LogMessage::SendToLog()
@ 0x7f9a445290 google::LogMessage::Flush()
@ 0x7f9a447eb4 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f9a6bae2c caffe::SyncedMemory::to_gpu()
@ 0x7f9a6b9eb4 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f9a6c94ac caffe::Blob<>::mutable_gpu_data()
@ 0x7f9a8d9f6c caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7f9a87dcf8 caffe::Net<>::ForwardFromTo()
@ 0x40afb0 warmup()
@ 0x40f72c processFrame()
@ 0x7f99a32fc4 start_thread
Aborted (core dumped)
Some one can help me out?

I'm not too much of an expert on this, but is it possible to use binarized neural networks to improve the performance of this project?

I have compiled successfully,but when running rtpose.bin, if command include --video -- write_frames, the process is killed, if --image_dir -write_frames ,the process is finished ,but nothing in output .why?
is there a demo like caffe/examples/cpp_classification.cpp?

Dear Authors,

First thank you very much for providing such a powerful tool!! It is really awesome!

However, when I estimated the skeletons in the KTH dataset, I found some inconsistencies in the generated json files. I could not fully understand why this problem happens.

For example, when I processed the video person11_handwaving_d2_uncomp.avi, in frame000156.json I got:

{
"version":0.1,
"bodies":[
{
"joints":[0,0,0,108.623,23.1522,0.252051,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,110.581,52.8297,0.351235,113.17,74.0262,0.420707,114.796,95.5481,0.566846,102.767,52.827,0.365775,104.713,73.6887,0.410267,100.82,95.221,0.55647,107.963,37.5057,0.421713]
},
{
"joints":[0,0,0,108.623,23.1522,0.252051,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,110.581,52.8297,0.351235,113.17,74.0262,0.420707,114.796,95.5481,0.566846,102.767,52.827,0.365775,104.713,73.6887,0.410267,100.82,95.221,0.55647,107.963,37.5057,0.421713]
},
{
"joints":[0,0,0,108.623,23.1522,0.252051,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,110.581,52.8297,0.351235,113.17,74.0262,0.420707,114.796,95.5481,0.566846,102.767,52.827,0.365775,104.713,73.6887,0.410267,100.82,95.221,0.55647,107.963,37.5057,0.421713]
}]
}

Here are my questions:
(1) What is the meaning of the 0 values in the "joints" entry? Do they imply detection failure?
(2) Only one person appears in the scene, so why three skeletons are generated??

I am looking forward to your answer, which are very important for my ongoing research; )

best,
Yan

Hi,

I've started compiling the project on OSX 10.11.5 with CUDA 7.5 installed and got this error:

examples/rtpose/rtpose.cpp:1088:22: error: variable length array of non-POD element type 'Frame'
    Frame frame_batch[BATCH_SIZE];

Additionally I'm getting a linking error against GOMP:

ld: library not found for -lgomp

It's confusing to see LD -o .build_release/lib/libcaffe.so.1.0.0-rc3 instead of something like LD -o .build_release/lib/libcaffe.dylib.1.0.0-rc3, even though ./build_releases/lib only contains libcaffe.a

What should I tweak to successfully link and compile?

This is the more verbose output:

------------------------- Compiling Caffe & CPM -------------------------
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
/usr/bin/clang++ -shared -o .build_release/lib/libcaffe.so.1.0.0-rc3 .build_release/src/caffe/proto/caffe.pb.o .build_release/src/caffe/blob.o .build_release/src/caffe/common.o .build_release/src/caffe/cpm/layers/imresize_layer.o .build_release/src/caffe/cpm/layers/nms_layer.o .build_release/src/caffe/cpm/util/math_functions.o .build_release/src/caffe/data_reader.o .build_release/src/caffe/data_transformer.o .build_release/src/caffe/internal_thread.o .build_release/src/caffe/layer.o .build_release/src/caffe/layer_factory.o .build_release/src/caffe/layers/absval_layer.o .build_release/src/caffe/layers/accuracy_layer.o .build_release/src/caffe/layers/argmax_layer.o .build_release/src/caffe/layers/base_conv_layer.o .build_release/src/caffe/layers/base_data_layer.o .build_release/src/caffe/layers/batch_norm_layer.o .build_release/src/caffe/layers/batch_reindex_layer.o .build_release/src/caffe/layers/bias_layer.o .build_release/src/caffe/layers/bnll_layer.o .build_release/src/caffe/layers/concat_layer.o .build_release/src/caffe/layers/contrastive_loss_layer.o .build_release/src/caffe/layers/conv_layer.o .build_release/src/caffe/layers/crop_layer.o .build_release/src/caffe/layers/cudnn_conv_layer.o .build_release/src/caffe/layers/cudnn_lcn_layer.o .build_release/src/caffe/layers/cudnn_lrn_layer.o .build_release/src/caffe/layers/cudnn_pooling_layer.o .build_release/src/caffe/layers/cudnn_relu_layer.o .build_release/src/caffe/layers/cudnn_sigmoid_layer.o .build_release/src/caffe/layers/cudnn_softmax_layer.o .build_release/src/caffe/layers/cudnn_tanh_layer.o .build_release/src/caffe/layers/data_layer.o .build_release/src/caffe/layers/deconv_layer.o .build_release/src/caffe/layers/dropout_layer.o .build_release/src/caffe/layers/dummy_data_layer.o .build_release/src/caffe/layers/eltwise_layer.o .build_release/src/caffe/layers/elu_layer.o .build_release/src/caffe/layers/embed_layer.o .build_release/src/caffe/layers/euclidean_loss_layer.o .build_release/src/caffe/layers/euclideanmask_loss_layer.o .build_release/src/caffe/layers/exp_layer.o .build_release/src/caffe/layers/filter_layer.o .build_release/src/caffe/layers/flatten_layer.o .build_release/src/caffe/layers/hdf5_data_layer.o .build_release/src/caffe/layers/hdf5_output_layer.o .build_release/src/caffe/layers/hinge_loss_layer.o .build_release/src/caffe/layers/im2col_layer.o .build_release/src/caffe/layers/image_data_layer.o .build_release/src/caffe/layers/infogain_loss_layer.o .build_release/src/caffe/layers/inner_product_layer.o .build_release/src/caffe/layers/input_layer.o .build_release/src/caffe/layers/log_layer.o .build_release/src/caffe/layers/loss_layer.o .build_release/src/caffe/layers/lrn_layer.o .build_release/src/caffe/layers/lstm_layer.o .build_release/src/caffe/layers/lstm_unit_layer.o .build_release/src/caffe/layers/maximum_layer.o .build_release/src/caffe/layers/memory_data_layer.o .build_release/src/caffe/layers/multinomial_logistic_loss_layer.o .build_release/src/caffe/layers/mvn_layer.o .build_release/src/caffe/layers/neuron_layer.o .build_release/src/caffe/layers/parameter_layer.o .build_release/src/caffe/layers/pooling_layer.o .build_release/src/caffe/layers/power_layer.o .build_release/src/caffe/layers/prelu_layer.o .build_release/src/caffe/layers/recurrent_layer.o .build_release/src/caffe/layers/reduction_layer.o .build_release/src/caffe/layers/relu_layer.o .build_release/src/caffe/layers/reshape_layer.o .build_release/src/caffe/layers/rnn_layer.o .build_release/src/caffe/layers/scale_layer.o .build_release/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_release/src/caffe/layers/sigmoid_layer.o .build_release/src/caffe/layers/silence_layer.o .build_release/src/caffe/layers/slice_layer.o .build_release/src/caffe/layers/softmax_layer.o .build_release/src/caffe/layers/softmax_loss_layer.o .build_release/src/caffe/layers/split_layer.o .build_release/src/caffe/layers/spp_layer.o .build_release/src/caffe/layers/tanh_layer.o .build_release/src/caffe/layers/threshold_layer.o .build_release/src/caffe/layers/tile_layer.o .build_release/src/caffe/layers/window_data_layer.o .build_release/src/caffe/net.o .build_release/src/caffe/parallel.o .build_release/src/caffe/solver.o .build_release/src/caffe/solvers/adadelta_solver.o .build_release/src/caffe/solvers/adagrad_solver.o .build_release/src/caffe/solvers/adam_solver.o .build_release/src/caffe/solvers/nesterov_solver.o .build_release/src/caffe/solvers/rmsprop_solver.o .build_release/src/caffe/solvers/sgd_solver.o .build_release/src/caffe/syncedmem.o .build_release/src/caffe/util/benchmark.o .build_release/src/caffe/util/blocking_queue.o .build_release/src/caffe/util/cudnn.o .build_release/src/caffe/util/db.o .build_release/src/caffe/util/db_leveldb.o .build_release/src/caffe/util/db_lmdb.o .build_release/src/caffe/util/hdf5.o .build_release/src/caffe/util/im2col.o .build_release/src/caffe/util/insert_splits.o .build_release/src/caffe/util/io.o .build_release/src/caffe/util/math_functions.o .build_release/src/caffe/util/signal_handler.o .build_release/src/caffe/util/upgrade_proto.o .build_release/src/gtest/gtest-all.o .build_release/src/rtpose/modelDescriptor.o .build_release/src/rtpose/modelDescriptorFactory.o .build_release/cuda/src/caffe/cpm/layers/imresize_layer.o .build_release/cuda/src/caffe/cpm/layers/nms_layer.o .build_release/cuda/src/caffe/cpm/util/math_functions.o .build_release/cuda/src/caffe/layers/absval_layer.o .build_release/cuda/src/caffe/layers/base_data_layer.o .build_release/cuda/src/caffe/layers/batch_norm_layer.o .build_release/cuda/src/caffe/layers/batch_reindex_layer.o .build_release/cuda/src/caffe/layers/bias_layer.o .build_release/cuda/src/caffe/layers/bnll_layer.o .build_release/cuda/src/caffe/layers/concat_layer.o .build_release/cuda/src/caffe/layers/contrastive_loss_layer.o .build_release/cuda/src/caffe/layers/conv_layer.o .build_release/cuda/src/caffe/layers/crop_layer.o .build_release/cuda/src/caffe/layers/cudnn_conv_layer.o .build_release/cuda/src/caffe/layers/cudnn_lcn_layer.o .build_release/cuda/src/caffe/layers/cudnn_lrn_layer.o .build_release/cuda/src/caffe/layers/cudnn_pooling_layer.o .build_release/cuda/src/caffe/layers/cudnn_relu_layer.o .build_release/cuda/src/caffe/layers/cudnn_sigmoid_layer.o .build_release/cuda/src/caffe/layers/cudnn_softmax_layer.o .build_release/cuda/src/caffe/layers/cudnn_tanh_layer.o .build_release/cuda/src/caffe/layers/deconv_layer.o .build_release/cuda/src/caffe/layers/dropout_layer.o .build_release/cuda/src/caffe/layers/eltwise_layer.o .build_release/cuda/src/caffe/layers/elu_layer.o .build_release/cuda/src/caffe/layers/embed_layer.o .build_release/cuda/src/caffe/layers/euclidean_loss_layer.o .build_release/cuda/src/caffe/layers/euclideanmask_loss_layer.o .build_release/cuda/src/caffe/layers/exp_layer.o .build_release/cuda/src/caffe/layers/filter_layer.o .build_release/cuda/src/caffe/layers/hdf5_data_layer.o .build_release/cuda/src/caffe/layers/hdf5_output_layer.o .build_release/cuda/src/caffe/layers/im2col_layer.o .build_release/cuda/src/caffe/layers/inner_product_layer.o .build_release/cuda/src/caffe/layers/log_layer.o .build_release/cuda/src/caffe/layerCXX examples/rtpose/rtpose.cpp
s/lrn_layer.o .build_release/cuda/src/caffe/layers/lstm_unit_layer.o .build_release/cuda/src/caffe/layers/maximum_layer.o .build_release/cuda/src/caffe/layers/mvn_layer.o .build_release/cuda/src/caffe/layers/pooling_layer.o .build_release/cuda/src/caffe/layers/power_layer.o .build_release/cuda/src/caffe/layers/prelu_layer.o .build_release/cuda/src/caffe/layers/recurrent_layer.o .build_release/cuda/src/caffe/layers/reduction_layer.o .build_release/cuda/src/caffe/layers/relu_layer.o .build_release/cuda/src/caffe/layers/scale_layer.o .build_release/cuda/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_release/cuda/src/caffe/layers/sigmoid_layer.o .build_release/cuda/src/caffe/layers/silence_layer.o .build_release/cuda/src/caffe/layers/slice_layer.o .build_release/cuda/src/caffe/layers/softmax_layer.o .build_release/cuda/src/caffe/layers/softmax_loss_layer.o .build_release/cuda/src/caffe/layers/split_layer.o .build_release/cuda/src/caffe/layers/tanh_layer.o .build_release/cuda/src/caffe/layers/threshold_layer.o .build_release/cuda/src/caffe/layers/tile_layer.o .build_release/cuda/src/caffe/solvers/adadelta_solver.o .build_release/cuda/src/caffe/solvers/adagrad_solver.o .build_release/cuda/src/caffe/solvers/adam_solver.o .build_release/cuda/src/caffe/solvers/nesterov_solver.o .build_release/cuda/src/caffe/solvers/rmsprop_solver.o .build_release/cuda/src/caffe/solvers/sgd_solver.o .build_release/cuda/src/caffe/util/im2col.o .build_release/cuda/src/caffe/util/math_functions.o .build_release/cuda/src/rtpose/renderFunctions.o -Wl,-install_name,@rpath/libcaffe.so.1.0.0-rc3 -Wl,-rpath,@loader_path/../../build/lib -pthread -fPIC -DCAFFE_VERSION=1.0.0-rc3 -DGTEST_USE_OWN_TR1_TUPLE=1 -DNDEBUG -O3 -DUSE_CUDNN -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I.build_release/src -I./src -I./include -I/usr/local/cuda/include -I/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers/ -Wall -Wno-sign-compare -Wno-unneeded-internal-declaration -fopenmp -Wl,-rpath, /usr/local/cuda/lib -framework Accelerate -L/usr/lib -L/usr/local/lib -L/usr/lib -L/usr/local/cuda/lib -L/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A -L.build_release/lib  -lcudart -lcublas -lcurand -lglog -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -lopencv_core -lopencv_highgui -lopencv_imgproc -lopencv_contrib -lboost_thread-mt -lcudnn -lcblas
/usr/bin/clang++ examples/rtpose/rtpose.cpp -MMD -MP -march=native -pthread -fPIC -DCAFFE_VERSION=1.0.0-rc3 -DGTEST_USE_OWN_TR1_TUPLE=1 -DNDEBUG -O3 -DUSE_CUDNN -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -I.build_release/src -I./src -I./include -I/usr/local/cuda/include -I/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers/ -Wall -Wno-sign-compare -Wno-unneeded-internal-declaration -std=c++11 -fopenmp -c -o .build_release/examples/rtpose/rtpose.o 2> .build_release/examples/rtpose/rtpose.o.warnings.txt \
		|| (cat .build_release/examples/rtpose/rtpose.o.warnings.txt; exit 1)
clang: warning: argument unused during compilation: '-pthread'
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1
make: *** Waiting for unfinished jobs....
examples/rtpose/rtpose.cpp:1088:22: error: variable length array of non-POD element type 'Frame'
    Frame frame_batch[BATCH_SIZE];
                     ^
1 error generated.
make: *** [.build_release/examples/rtpose/rtpose.o] Error 1

This is the configuration I've used to compile, tweaked from the repo's Ubuntu 14 one:
Makefile.config.OSX.10.11.5.example:

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_50,code=compute_50 \
		-gencode arch=compute_52,code=sm_52 \
		# -gencode arch=compute_60,code=sm_60 \
		# -gencode arch=compute_61,code=sm_61
# Deprecated
#CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
#		-gencode arch=compute_20,code=sm_21 \
#		-gencode arch=compute_30,code=sm_30 \
#		-gencode arch=compute_35,code=sm_35 \
#		-gencode arch=compute_50,code=sm_50 \
#		-gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
BLAS_INCLUDE := /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers/
BLAS_LIB := /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		# $(ANACONDA_HOME)/include/python2.7 \
		# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
# INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
# LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
# Q ?= @

and the install_caffe_and_cpm_osx.sh script:

#!/bin/bash



echo "------------------------- INSTALLING CAFFE AND CPM -------------------------"
echo "NOTE: This script assumes that CUDA and cuDNN are already installed on your machine. Otherwise, it might fail."



function exitIfError {
    if [[ $? -ne 0 ]] ; then
        echo ""
        echo "------------------------- -------------------------"
        echo "Errors detected. Exiting script. The software might have not been successfully installed."
        echo "------------------------- -------------------------"
        exit 1
    fi
}



echo "------------------------- Checking Number of Processors -------------------------"
NUM_CORES=$(grep -c ^processor /proc/cpuinfo 2>/dev/null || sysctl -n hw.ncpu)
echo "$NUM_CORES cores"
exitIfError
echo "------------------------- Number of Processors Checked -------------------------"
echo ""



echo "------------------------- Installing some Caffe Dependencies -------------------------"
# Basic
# sudo apt-get --assume-yes update
# sudo apt-get --assume-yes install build-essential
#General dependencies
brew install protobuf leveldb snappy hdf5
# with Python pycaffe needs dependencies built from source - from http://caffe.berkeleyvision.org/install_osx.html
# brew install --build-from-source --with-python -vd protobuf
# brew install --build-from-source -vd boost boost-python
# without Python the usual installation suffices
brew install boost
# sudo apt-get --assume-yes install libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler
# sudo apt-get --assume-yes install --no-install-recommends libboost-all-dev
# Remaining dependencies, 14.04
brew install gflags glog lmdb
# if [[ $ubuntu_le_14 == true ]]; then
#     sudo apt-get --assume-yes install libgflags-dev libgoogle-glog-dev liblmdb-dev
# fi
# OpenCV 2.4
# sudo apt-get --assume-yes install libopencv-dev
exitIfError
echo "------------------------- Some Caffe Dependencies Installed -------------------------"
echo ""



echo "------------------------- Compiling Caffe & CPM -------------------------"
cp Makefile.config.OSX.10.11.5.example Makefile.config
make all -j$NUM_CORES
# make test -j$NUM_CORES
# make runtest -j$NUM_CORES
exitIfError
echo "------------------------- Caffe & CPM Compiled -------------------------"
echo ""



# echo "------------------------- Installing CPM -------------------------"
# echo "Compiled"
# exitIfError
# echo "------------------------- CPM Installed -------------------------"
# echo ""



echo "------------------------- Downloading CPM Models -------------------------"
models_folder="./model/"
# COCO
coco_folder="$models_folder"coco/""
coco_model="$coco_folder"pose_iter_440000.caffemodel""
if [ ! -f $coco_model ]; then
    wget http://posefs1.perception.cs.cmu.edu/Users/tsimon/Projects/coco/data/models/coco/pose_iter_440000.caffemodel -P $coco_folder
fi
exitIfError
# MPI
mpi_folder="$models_folder"mpi/""
mpi_model="$mpi_folder"pose_iter_160000.caffemodel""
if [ ! -f $mpi_model ]; then
    wget http://posefs1.perception.cs.cmu.edu/Users/tsimon/Projects/coco/data/models/mpi/pose_iter_160000.caffemodel -P $mpi_folder
fi
exitIfError
echo "Models downloaded"
echo "------------------------- CPM Models Downloaded -------------------------"
echo ""



echo "------------------------- CAFFE AND CPM INSTALLED -------------------------"
echo ""

Thank you,
George

[platform]Ubuntu 14.04+cuda8.0 +gtx1060 6G
when I excute the command of rtpose.bin , show infomation as follows:
forever@GoogleMind:~/caffe_demo-master$ ./build/examples/rtpose/rtpose.bin --video test.mp4 --logtostderr

I1230 17:57:56.720556 11047 rtpose.cpp:1711] Display resolution: 1280x720
I1230 17:57:56.720675 11047 rtpose.cpp:1717] Net resolution: 656x368
I1230 17:57:56.720731 11047 rtpose.cpp:1484] Finish spawning 1 threads.
I1230 17:57:56.720790 11050 rtpose.cpp:175] Setting GPU 0
init done
opengl support available
I1230 17:57:56.964375 11050 rtpose.cpp:180] GPU 0: copying to person net
I1230 17:58:00.892221 11050 rtpose.cpp:202] start_scale = 1
I1230 17:58:00.892299 11050 rtpose.cpp:231] Dry running...
I1230 17:58:01.170068 11050 rtpose.cpp:233] Success.
I1230 17:58:01.170305 11050 rtpose.cpp:1081] GPU 0 is ready
I1230 17:58:30.679666 11062 rtpose.cpp:540] Looping video after 615 frames
I1230 17:59:04.849298 11062 rtpose.cpp:540] Looping video after 615 frames
I1230 17:59:39.570911 11062 rtpose.cpp:540] Looping video after 615 frames

After 10 minutes,this program causes the system to crash.
I could understand why?

Hi everyone,

Thank you for sharing this amazing work!

I am opening this issue because I am experiencing a sporadic GPU out of memory error.

It was not weird to see it the first time, because I am using a laptop with a NVidia Quadro K1100M which has only 2GB RAM.

Nevertheless, the weird thing is that after a log-out/log-in the error disappears letting me run the rtpose binary (even with the resolution which should ask for 3GB RAM). Do you have an explanation for this? How could I prevent to log-out/log-in every time I am seeing the error?

I am running the rtpose demo on a Linux Mint 17.3 (derived from Ubuntu 14.04). Here is the stack trace:

F0123 10:43:52.732602  6844 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
    @     0x7fabd153fdaa  (unknown)
    @     0x7fabd153fce4  (unknown)
    @     0x7fabd153f6e6  (unknown)
    @     0x7fabd1542687  (unknown)
    @     0x7fabd1d1f962  caffe::SyncedMemory::to_gpu()
    @     0x7fabd1d1ecd9  caffe::SyncedMemory::mutable_gpu_data()
    @     0x7fabd1cfb1a2  caffe::Blob<>::mutable_gpu_data()
    @     0x7fabd1db0080  caffe::NmsLayer<>::Forward_gpu()
    @     0x7fabd1d2cc1b  caffe::Net<>::ForwardFromTo()
    @           0x409bcf  warmup()
    @           0x40fc2b  processFrame()
    @     0x7fabcf9ac184  start_thread
    @     0x7fabcf6d937d  (unknown)
    @              (nil)  (unknown)
Aborted

thanks for your share, but when I ran 'install_caffe_and_cpm.sh', It has some problem, following:
NVCC src/caffe/util/im2col.cu
nvcc fatal : Unsupported gpu architecture 'compute_60'
make: *** [.build_release/cuda/src/caffe/util/im2col.o] Error 1
make: *** Waiting for unfinished jobs....
which cudnn and cuda do you use? in my case I use cudnn4 and cuda7.5
Or do you know the problem? other way to solve it? thank you

Hi,

Thanks for sharing your code!

After I made it and tried to run the demo, I got an OOM error. I tried several video and pictures and the results are the same. That's really wired since the Titan X has a cuda memory of 12GB. What's the memory size it need for running the demo?

Thanks

I have compiled caffe_rtpose successfully on my following enviroment.

• using nvidia-docker
• ubuntu 16.04
• cuda 8.0
• cudnn 5.1.10
• opencv 3.1

This enviroment is built with following Dockerfile.

FROM nvidia/cuda:8.0-devel-ubuntu16.04
LABEL maintainer "NVIDIA CORPORATION <[email protected]>"

RUN mkdir OpenCV && cd OpenCV

RUN apt-get update && apt-get install -y \
  build-essential \
  checkinstall \
  cmake \
  pkg-config \
  yasm \
  libtiff5-dev \
  libjpeg-dev \
  libjasper-dev \
  libavcodec-dev \
  libavformat-dev \
  libswscale-dev \
  libdc1394-22-dev \
  libgstreamer0.10-dev \
  libgstreamer-plugins-base0.10-dev \
  libv4l-dev \
  python-dev \
  python-numpy \
  python-pip \
  libtbb-dev \
  libeigen3-dev \
  libqt4-dev \
  libgtk2.0-dev \
  # Doesn't work libfaac-dev \
  libmp3lame-dev \
  libopencore-amrnb-dev \
  libopencore-amrwb-dev \
  libtheora-dev \
  libvorbis-dev \
  libxvidcore-dev \
  x264 \ 
  v4l-utils \
  libgtk2.0-dev \
  unzip \
  libhdf5-dev \
  wget \
  curl \
  sudo \
  git \
  vim \
  lsb-release \
  libopenblas-dev \
  libatlas-base-dev \
  libgflags-dev \
  libgoogle-glog-dev \
  liblmdb-dev


ENV CUDNN_VERSION 5.1.10
LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}"
RUN CUDNN_DOWNLOAD_SUM=c10719b36f2dd6e9ddc63e3189affaa1a94d7d027e63b71c3f64d449ab0645ce && \
    curl -fsSL http://developer.download.nvidia.com/compute/redist/cudnn/v5.1/cudnn-8.0-linux-x64-v5.1.tgz -O && \
    echo "$CUDNN_DOWNLOAD_SUM  cudnn-8.0-linux-x64-v5.1.tgz" | sha256sum -c --strict - && \
    tar -xzf cudnn-8.0-linux-x64-v5.1.tgz -C /usr/local && \
    rm cudnn-8.0-linux-x64-v5.1.tgz && \
    ldconfig


RUN cd /opt && \
  wget https://github.com/daveselinger/opencv/archive/3.1.0-with-cuda8.zip -O opencv-3.1.0.zip -nv && \
  unzip opencv-3.1.0.zip && \
  mv opencv-3.1.0-with-cuda8 opencv-3.1.0 && \
  cd opencv-3.1.0 && \
  rm -rf build && \
  mkdir build && \
  cd build && \
  cmake -D CUDA_ARCH_BIN=3.2 \
    -D CUDA_ARCH_PTX=3.2 \
    -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D WITH_TBB=ON \
    -D BUILD_NEW_PYTHON_SUPPORT=ON \
    -D WITH_V4L=ON \
    -D BUILD_TIFF=ON \
    -D WITH_QT=ON \
    -D ENABLE_PRECOMPILED_HEADERS=OFF \
    -D WITH_OPENGL=ON .. && \
  make -j4 && \
  make install && \
  echo "/usr/local/lib" | sudo tee -a /etc/ld.so.conf.d/opencv.conf && \
  ldconfig
RUN cp /opt/opencv-3.1.0/build/lib/cv2.so /usr/lib/python2.7/dist-packages/cv2.so

I confirmed this docker container can use webcam and GUI, and I try to run following command, but nothing happens... X(

$ ./build/examples/rtpose/rtpose.bin 
init done 
opengl support available 

$ ./build/examples/rtpose/rtpose.bin --video ./build/examples/rtpose/test.mp4 --num_gpu 1 --write_frames output/
init done 
opengl support available 

user1@multilab:~/YALI/RMPE/caffe$ ./build/examples/rtpose/rtpose.bin --video dance.mp4 --net_resolution 656x368 --num_gpu 2 --logtostderr --write_frames output/
I0124 16:48:03.524648 28526 rtpose.cpp:1687] Display resolution: 1280x720
I0124 16:48:03.524807 28526 rtpose.cpp:1693] Net resolution: 656x368
I0124 16:48:03.524914 28526 rtpose.cpp:1468] Finish spawning 2 threads.
I0124 16:48:03.525133 28538 rtpose.cpp:175] Setting GPU 0
I0124 16:48:03.525133 28539 rtpose.cpp:175] Setting GPU 1
init done
opengl support available
I0124 16:48:04.400218 28538 rtpose.cpp:180] GPU 0: copying to person net
I0124 16:48:08.809712 28538 rtpose.cpp:202] start_scale = 1
I0124 16:48:08.809803 28538 rtpose.cpp:231] Dry running...
I0124 16:48:08.810757 28539 rtpose.cpp:202] start_scale = 1
I0124 16:48:08.810843 28539 rtpose.cpp:231] Dry running...
I0124 16:48:09.038563 28539 rtpose.cpp:233] Success.
I0124 16:48:09.038861 28539 rtpose.cpp:1081] GPU 1 is ready
I0124 16:48:09.041012 28538 rtpose.cpp:233] Success.
I0124 16:48:09.042523 28538 rtpose.cpp:1081] GPU 0 is ready
I0124 16:50:39.920331 28634 rtpose.cpp:526] Done, exiting. # frames: 4549

I ran it like this , but nothing in the 'output' file, and it seems it doesn't work. Do you know the problem?
thank you

I'm trying to get the forked version of caffe to compile, but am unable to pass the tests or run the model. In either case, I have the same error: Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

For reference, I'm on Ubuntu 16.04.01. I've tried a couple of different cudnn/cuda configs but seem to get that error each time. I've also successfully built a couple of other caffe / forked caffe versions on this machine, without seeing this error.

Anyways, for reference, what versions of cuda/cudnn have been successful? Thanks!

Is it possible to make this project workable on specs specified in title?
I have read at [1] that their will be a version released for 2GB VGAs. When will it be released?
[1] - #18 (comment)

We compiled and executed the rtpose.bin (w/o making any changes) and for some reason, its not generating any output for either image or video file.

There is no error either.
I have pasted the command and output below

Any idea why it is not generating json file?

Thanks

./build/examples/rtpose/rtpose.bin --image_dir /home/xxx/images/dnn/ --write_json output
init done
opengl support available
E0109 19:00:27.296622 118173 rtpose.cpp:1538] rtcpm successfully finished.
E0109 19:00:27.296898 118173 rtpose.cpp:1541] Total time: 4.61036 seconds.
xxx@yyy:~/project/dnn/caffe_rtpose$ ls output/

I try to run it with a 13 second video but seems like it is stuck in a infinite loop.

.build_release/examples/rtpose/rtpose.bin --video ../data/videos/VID00002.MP4 --write_json ../data-json/ --num_scales 3 --scale_gap 0.15 --logtostderr

I0401 14:02:38.742015  2667 rtpose.cpp:1692] Display resolution: 1280x720
I0401 14:02:38.742077  2667 rtpose.cpp:1698] Net resolution: 656x368
I0401 14:02:38.742128  2667 rtpose.cpp:1473] Finish spawning 1 threads.
I0401 14:02:38.742218  2669 rtpose.cpp:176] Setting GPU 0
init done
opengl support available
I0401 14:02:38.978080  2669 rtpose.cpp:181] GPU 0: copying to person net
I0401 14:02:39.859575  2669 rtpose.cpp:203] start_scale = 1
I0401 14:02:39.859652  2669 rtpose.cpp:232] Dry running...
I0401 14:02:41.381358  2669 rtpose.cpp:234] Success.
I0401 14:02:41.381680  2669 rtpose.cpp:1082] GPU 0 is ready
I0401 14:02:56.365974  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:03:10.640458  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:03:28.184327  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:03:44.831750  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:04:04.101912  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:04:22.384856  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:04:41.226457  2684 rtpose.cpp:541] Looping video after 419 frames
I0401 14:05:00.453107  2684 rtpose.cpp:541] Looping video after 419 frames

Any one come across with this?

It's much easier to use this repo from docker, but how to run video/image or web camera from docker command line?

I saw caffe training here:
https://github.com/CMU-Perceptual-Computing-Lab/caffe_rtpose/tree/master/docker

I want to make pose recognition similar to real time. To do this,

I used this expression below.

--num_gpu 4 <--- Parallelize over this number of GPUs. Default is 1.

and I got this error.
F1219 15:59:41.175240 36602 common.cpp:137] Check failed: error == cudaSuccess (10 vs. 0) invalid device ordinal

I just have one graphic card, Do I have to increase the number of graphic cards to recognize the human pose in real time??

Hi,
I hope to get smaller models. Can you provide training codes? Or can you give me some new models? Especially those with small numbers of layers. I mean those basic conv layers which extract the foundational features. eg: conv1_1, conv1_2....
It seems they are greatly affect the speed of the network.

WARNING: Logging before InitGoogleLogging() is written to STDERR
F0317 17:32:11.124943 20749 solver_factory.hpp:67] Check failed: registry.count(type) == 0 (1 vs. 0) Solver type Nesterov already registered.
*** Check failure stack trace: ***
Aborted (core dumped)

Thank you for sharing the source code.

I have the same problem with issue #7 with OpenCV version 2.4.8.

`hzz@bdp-gpu-01:~/MultiPose/caffe_rtpose$ ./build/examples/rtpose/rtpose.bin --no_display --image_dir image/ --write_frames output/

E0307 17:48:00.193135 16291 rtpose.cpp:1543] rtcpm successfully finished.

E0307 17:48:00.193334 16291 rtpose.cpp:1546] Total time: 10.2573 seconds.

hzz@bdp-gpu-01:~/MultiPose/caffe_rtpose$ cd output/

hzz@bdp-gpu-01:~/MultiPose/caffe_rtpose/output$ ls

hzz@bdp-gpu-01:~/MultiPose/caffe_rtpose/output$`

I have turned off the display but there is still no output.

Does anyone know how I can fix this?

./build/examples/rtpose/rtpose.bin --help

rtpose.bin: Warning: SetUsageMessage() never called

  Flags from /build/gflags-YYnfS9/gflags-2.1.2/src/gflags.cc:
    -flagfile (load flags from file) type: string default: ""
    -fromenv (set flags from the environment [use 'export FLAGS_flag1=value'])
      type: string default: ""
    -tryfromenv (set flags from the environment if present) type: string
      default: ""
    -undefok (comma-separated list of flag names that it is okay to specify on
      the command line even if the program does not define a flag with that
      name.  IMPORTANT: flags in this list that have arguments MUST use the
      flag=value format) type: string default: ""

  Flags from /build/gflags-YYnfS9/gflags-2.1.2/src/gflags_completions.cc:
    -tab_completion_columns (Number of columns to use in output for tab
      completion) type: int32 default: 80
    -tab_completion_word (If non-empty, HandleCommandLineCompletions() will
      hijack the process and attempt to do bash-style command line flag
      completion on this value.) type: string default: ""

  Flags from /build/gflags-YYnfS9/gflags-2.1.2/src/gflags_reporting.cc:
    -help (show help on all flags [tip: all flags can have two dashes])
      type: bool default: false currently: true
    -helpfull (show help on all flags -- same as -help) type: bool
      default: false
    -helpmatch (show help on modules whose name contains the specified substr)
      type: string default: ""
    -helpon (show help on the modules named by this flag value) type: string
      default: ""
    -helppackage (show help on all modules in the main package) type: bool
      default: false
    -helpshort (show help on only the main module for this program) type: bool
      default: false
    -helpxml (produce an xml version of help) type: bool default: false
    -version (show version and build info and exit) type: bool default: false



  Flags from examples/rtpose/rtpose.cpp:
    -caffemodel (Caffe model.) type: string
      default: "model/coco/pose_iter_440000.caffemodel"
    -caffeproto (Caffe deploy prototxt.) type: string
      default: "model/coco/pose_deploy_linevec.prototxt"
    -camera (The camera index for VideoCapture.) type: int32 default: 0
    -camera_resolution (Size of the camera frames to ask for.) type: string
      default: "1280x720"
    -fullscreen (Run in fullscreen mode (press f during runtime to toggle))
      type: bool default: false
    -image_dir (Process a directory of images.) type: string default: ""
    -net_resolution (Multiples of 16.) type: string default: "656x368"
    -no_display (Do not open a display window.) type: bool default: false
    -no_frame_drops (Dont drop frames.) type: bool default: false
    -no_text (Do not write text on output images.) type: bool default: false
    -num_gpu (The number of GPU devices to use.) type: int32 default: 1
    -num_scales (Number of scales to average) type: int32 default: 1
    -part_to_show (Part to show from the start.) type: int32 default: 0
    -resolution (The image resolution (display).) type: string
      default: "1280x720"
    -scale_gap (Scale gap between scales. No effect unless num_scales>1)
      type: double default: 0.29999999999999999
    -start_device (GPU device start number.) type: int32 default: 0
    -start_frame (Skip to frame # of video) type: int32 default: 0
    -start_scale (Initial scale. Must cv::Match net_resolution) type: double
      default: 1
    -video (Use a video file instead of the camera.) type: string default: ""
    -write_frames (Write frames with format prefix%06d.jpg) type: string
      default: ""
    -write_json (Write joint data with json format as prefix%06d.json)
      type: string default: ""



  Flags from src/logging.cc:
    -alsologtoemail (log messages go to these email addresses in addition to
      logfiles) type: string default: ""
    -alsologtostderr (log messages go to stderr in addition to logfiles)
      type: bool default: false
    -colorlogtostderr (color messages logged to stderr (if supported by
      terminal)) type: bool default: false
    -drop_log_memory (Drop in-memory buffers of log contents. Logs can grow
      very quickly and they are rarely read before they need to be evicted from
      memory. Instead, drop them from memory as soon as they are flushed to
      disk.) type: bool default: true
    -log_backtrace_at (Emit a backtrace when logging at file:linenum.)
      type: string default: ""
    -log_dir (If specified, logfiles are written into this directory instead of
      the default logging directory.) type: string default: ""
    -log_link (Put additional links to the log files in this directory)
      type: string default: ""
    -log_prefix (Prepend the log prefix to the start of each log line)
      type: bool default: true
    -logbuflevel (Buffer log messages logged at this level or lower (-1 means
      don't buffer; 0 means buffer INFO only; ...)) type: int32 default: 0
    -logbufsecs (Buffer log messages for at most this many seconds) type: int32
      default: 30
    -logemaillevel (Email log messages logged at this level or higher (0 means
      email all; 3 means email FATAL only; ...)) type: int32 default: 999
    -logmailer (Mailer used to send logging email) type: string
      default: "/bin/mail"
    -logtostderr (log messages go to stderr instead of logfiles) type: bool
      default: false
    -max_log_size (approx. maximum log file size (in MB). A value of 0 will be
      silently overridden to 1.) type: int32 default: 1800
    -minloglevel (Messages logged at a lower level than this don't actually get
      logged anywhere) type: int32 default: 0
    -stderrthreshold (log messages at or above this level are copied to stderr
      in addition to logfiles.  This flag obsoletes --alsologtostderr.)
      type: int32 default: 2
    -stop_logging_if_full_disk (Stop attempting to log to disk if the disk is
      full.) type: bool default: false

  Flags from src/utilities.cc:
    -symbolize_stacktrace (Symbolize the stack trace in the tombstone)
      type: bool default: true

  Flags from src/vlog_is_on.cc:
    -v (Show all VLOG(m) messages for m <= this. Overridable by --vmodule.)
      type: int32 default: 0
    -vmodule (per-module verbose level. Argument is a comma-separated list of
      <module name>=<log level>. <module name> is a glob pattern, matched
      against the filename base (that is, name ignoring .cc/.h./-inl.h). <log
      level> overrides any value given by --v.) type: string default: ""

[platform] Ubuntu 14.04 +cuda8.0+cudnn5.1+opencv2.4.13
when I make all,it appears some problems:
1.std::runtime_error is not member of std,which could be solved by insert the header of stdexcept,then it will be make successfully!
2.After make all,when I run the command of "./build/examples/rtpose/rtpose.bin",it appear a error about threads, which is:
E1230 09:52:39.072165 3477 rtpose.cpp:1507] Finish spawning 1 threads. now waiting.
init done
opengl support available
However,then it all always waiting,no results,could anyone help me solver the problem.Thank you!

Hi,

I followed the steps described in the guide and I got this error when trying to build matcaffe:

'/Realtime_Multi-Person_Pose_Estimation/caffe_demo/matlab/+caffe/private/caffe_.mexa64': /Realtime_Multi-Person_Pose_Estimation/caffe_demo/matlab/+caffe/private/caffe_.mexa64: undefined symbol: _ZN2cv11getTextSizeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEidiPi

Any solution?

Thank you.

I downloaded the caffe_rtpose without modification and tried to use it to re-train the openpose model, however, I got the error:
Error parsing text-format caffe.NetParameter: 11:23: Message type "caffe.LayerParameter" has no field named "cpm_transform_param".
F0416 19:41:44.973068 11790 upgrade_proto.cpp:79] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: pose_train_test.prototxt

Please advise how to work around it

Thanks

i found that the implemention of cpu and gpu forward function of nms_layer is different

Hi,
Excellent work!
I want to render only the skeleton without the original image. Could this be possible?
Much thanks!

[lfx@gpu-com caffe_rtpose-master]$ ./build/examples/rtpose/rtpose.bin --image_dir ~/caffe_rtpose-master/image/
./build/examples/rtpose/rtpose.bin: /home/lfx/tools/anaconda2/lib/libtiff.so.5: no version information available (required by /usr/local/lib/libopencv_imgcodecs.so.3.1)
F0402 10:24:15.074151 25048 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f7d4a812e6d (unknown)
@ 0x7f7d4a814ced (unknown)
@ 0x7f7d4a812a5c (unknown)
@ 0x7f7d4a81563e (unknown)
@ 0x7f7d519eaed2 caffe::SyncedMemory::to_gpu()
@ 0x7f7d519ea249 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f7d51863e62 caffe::Blob<>::mutable_gpu_data()
@ 0x7f7d51a60e7f caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7f7d519b2f2b caffe::Net<>::ForwardFromTo()
@ 0x40bdff warmup()
@ 0x411e3b processFrame()
@ 0x7f7d3eb95dc5 start_thread
@ 0x7f7d3e8c473d __clone
Aborted (core dumped)

[lfx@gpu-com caffe_rtpose-master]$ nvidia-smi
Sun Apr 2 10:25:18 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26 Driver Version: 375.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GT 730 Off | 0000:01:00.0 N/A | N/A |
| 40% 59C P8 N/A / N/A | 2MiB / 978MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX TIT... Off | 0000:06:00.0 Off | N/A |
| 53% 83C P2 184W / 250W | 4192MiB / 12206MiB | 97% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
| 1 12602 C python 139MiB |
| 1 22669 C python 139MiB |
| 1 23028 C /home/xhq/caffe/build/tools/caffe 3903MiB |
+-----------------------------------------------------------------------------+

I have no idea why still have the error "out of memory" even test in just 1 image.
[lfx@gpu-com caffe_rtpose-master]$ ls image
image1.jpg

Hi,

I am getting the following error while testing it with video. Please help.

F0427 11:36:23.180558 8473 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f8f2bad15cd google::LogMessage::Fail()
@ 0x7f8f2bad3433 google::LogMessage::SendToLog()
@ 0x7f8f2bad115b google::LogMessage::Flush()
@ 0x7f8f2bad3e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f8f2f88cf40 caffe::SyncedMemory::to_gpu()
@ 0x7f8f2f88bf29 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f8f2f88e382 caffe::Blob<>::mutable_gpu_data()
@ 0x7f8f2f7a79b8 caffe::BaseConvolutionLayer<>::forward_gpu_gemm()
@ 0x7f8f2f93f796 caffe::ConvolutionLayer<>::Forward_gpu()
@ 0x7f8f2f8788e2 caffe::Net<>::ForwardFromTo()
@ 0x40a81b warmup()
@ 0x411204 processFrame()
@ 0x7f8f2a9d26ba start_thread
@ 0x7f8f2a70882d clone
@ (nil) (unknown)
Aborted (core dumped)

i am using:
./build/examples/rtpose/rtpose.bin --camera 0 --num_gpu 1 --write_frames output13/ --write_json output13/
only first frame is written to file, but i see subsequent frames being processed on the screen

Thanks for your share.
Do you use opencv2? Mine is opencv3, so it has problem?

I am integrating the rtpose code into my caffe using the instructions provided in the readme. I am almost there but am running into the following issue:

CXX/LD -o .build_release/examples/rtpose/rtpose.bin
.build_release/examples/rtpose/rtpose.o: In function `distanceThresholdPeaks(float const*, int, float*, ModelDescriptor*)':
rtpose.cpp:(.text+0xada): undefined reference to `ModelDescriptor::get_number_parts()'
.build_release/examples/rtpose/rtpose.o: In function `render(int, float*)':
rtpose.cpp:(.text+0x2357): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x2375): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x24b4): undefined reference to `render_mpi_parts(float*, int, int, int, int, float*, int, float*, float*, std::vector<int, std::allocator<int> >, int)'
rtpose.cpp:(.text+0x24fd): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x2528): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x255e): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x25d0): undefined reference to `render_coco_aff(float*, int, int, int, int, float*, int, float*, float*, std::vector<int, std::allocator<int> >, int, int)'
rtpose.cpp:(.text+0x2658): undefined reference to `render_coco_parts(float*, int, int, int, int, float*, int, float*, float*, std::vector<int, std::allocator<int> >, int, bool)'
.build_release/examples/rtpose/rtpose.o: In function `warmup(int)':
rtpose.cpp:(.text+0x3df7): undefined reference to `ModelDescriptorFactory::createModelDescriptor(ModelDescriptorFactory::Type, std::unique_ptr<ModelDescriptor, std::default_delete<ModelDescriptor> >&)'
rtpose.cpp:(.text+0x3fa7): undefined reference to `ModelDescriptorFactory::createModelDescriptor(ModelDescriptorFactory::Type, std::unique_ptr<ModelDescriptor, std::default_delete<ModelDescriptor> >&)'
.build_release/examples/rtpose/rtpose.o: In function `displayFrame(void*)':
rtpose.cpp:(.text+0x48ad): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x48fa): undefined reference to `ModelDescriptor::get_part_name[abi:cxx11](int)'
rtpose.cpp:(.text+0x4b58): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x51a2): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x51e5): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x5218): undefined reference to `ModelDescriptor::get_part_name[abi:cxx11](int)'
.build_release/examples/rtpose/rtpose.o: In function `connectLimbsCOCO(std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, float const*, float const*, int, float*, ModelDescriptor*)':
rtpose.cpp:(.text+0x72ee): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x7301): undefined reference to `ModelDescriptor::get_limb_sequence()'
rtpose.cpp:(.text+0x7319): undefined reference to `ModelDescriptor::get_map_idx()'
rtpose.cpp:(.text+0x7331): undefined reference to `ModelDescriptor::number_limb_sequence()'
.build_release/examples/rtpose/rtpose.o: In function `connectLimbs(std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >&, std::vector<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > >, std::allocator<std::vector<std::vector<double, std::allocator<double> >, std::allocator<std::vector<double, std::allocator<double> > > > > >&, float const*, float const*, int, float*, ModelDescriptor*)':
rtpose.cpp:(.text+0x8c03): undefined reference to `ModelDescriptor::get_number_parts()'
rtpose.cpp:(.text+0x8c16): undefined reference to `ModelDescriptor::get_limb_sequence()'
rtpose.cpp:(.text+0x8c2e): undefined reference to `ModelDescriptor::get_map_idx()'
rtpose.cpp:(.text+0x8c46): undefined reference to `ModelDescriptor::number_limb_sequence()'
collect2: error: ld returned 1 exit status
Makefile:630: recipe for target '.build_release/examples/rtpose/rtpose.bin' failed
make: *** [.build_release/examples/rtpose/rtpose.bin] Error 1

I had set the `CXXFLAG to -std=c++11, otherwise it would throw more errors. I am on Ubuntu 16.04 and using g++ version 5.4.

Any pointers on what might be going wrong?

Isn't multiplying by 16 and dividing by 16 equal to not doing anything?

Hi Everyone,

I am opening this issue because I have some problems in running rtpose using MPI model instead of COCO.

I am running the demo on an Ubuntu 14.04 machine. The demo runs fine using the COCO model.

This is what I get:
F0125 17:44:29.373692 7559 upgrade_proto.cpp:86] Check failed: ReadProtoFromBinaryFile(param_file, param) Failed to parse NetParameter file: /home/marco/workspace/caffe_rtpose/model/mpi/pose_iter_160000.caffemodel *** Check failure stack trace: *** @ 0x7f9b7c44cdaa (unknown) @ 0x7f9b7c44cce4 (unknown) @ 0x7f9b7c44c6e6 (unknown) @ 0x7f9b7c44f687 (unknown) @ 0x7f9b7c990eae caffe::ReadNetParamsFromBinaryFileOrDie() @ 0x7f9b7c9dc057 caffe::Net<>::CopyTrainedLayersFromBinaryProto() @ 0x7f9b7c9dc0cf caffe::Net<>::CopyTrainedLayersFrom() @ 0x44c3d8 warmup() @ 0x45455b processFrame() @ 0x7f9b7d74b184 start_thread @ 0x7f9b7b95137d (unknown) @ (nil) (unknown)

Am I missing something?

The error is like
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::lock_error >'
pure virtual method called
terminate called recursively

Please help me!

CMake Error at CMakeLists.txt:74 (add_subdirectory):
The source directory
D:/openpose/caffe_rtpose-master/examples
does not contain a CMakeLists.txt file.

I am not familiar with C++，and I want to get the information that has been detected(about the joints) to do something in realtime detection. I know the '--write_json',and I want to know where is the # code to implement this instruction.

Thanks a lot.

Hi,

When running rtpose.bin I'm running out of GPU memory.

116 syncedmem.cpp:64] Check failed: error == cudaSuccess (2 vs. 0) out of memory

Is there any way to decrease the amount of memory used or is the answer to use a GPU with more memory?

Cheers,
Tom

I compiled caffe window version on window environment but I don't know it is possible to compile caffe_rtpose. Could you give me some advises about this problem?

I am using GTX960M GPU and I compiled successfully by
./install_caffe_and_cpm.sh

However, after the build was done , and I want to test it by typing
./build/examples/rtpose/rtpose.bin

I failed it and I didn't know what's reason yet, someone can tell me?

Thanks
Bill
The following is the hard copy of running message:

./build/examples/rtpose/rtpose.bin
F0424 10:12:09.817428 1511 cudnn.hpp:113] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
@ 0x7fb59a3855cd google::LogMessage::Fail()
@ 0x7fb59a387433 google::LogMessage::SendToLog()
@ 0x7fb59a38515b google::LogMessage::Flush()
@ 0x7fb59a387e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7fb59aac38f8 caffe::CuDNNConvolutionLayer<>::Reshape()
@ 0x7fb59a9dad2e caffe::Net<>::Init()
@ 0x7fb59a9dd090 caffe::Net<>::Net()
@ 0x40a608 warmup()
@ 0x411084 processFrame()
@ 0x7fb5987886ba start_thread
@ 0x7fb5984be82d clone
@ (nil) (unknown)
Aborted (core dumped)

I am trying to compile it on a CentOS 6.5 machine, with all the requirements present. I get a bunch of assembler errors. I am able to compile the current BVLC/caffe source (master) on the same machine though.

.
.
.
/tmp/cctOfDIJ.s:48580: Error: no such instruction: `shlx %rax,%r8,%rcx'                                                                                                                                                              [37/1930]
/tmp/cctOfDIJ.s:48593: Error: no such instruction: `shlx %rax,%rdi,%r12'
/tmp/cctOfDIJ.s:48610: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/cctOfDIJ.s:48654: Error: no such instruction: `shlx %rbx,%rdi,%r8'
/tmp/cctOfDIJ.s:48656: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/cctOfDIJ.s:48696: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/cctOfDIJ.s:48727: Error: no such instruction: `shlx %rax,%r8,%r9'
/tmp/cctOfDIJ.s:48740: Error: no such instruction: `shlx %rax,%r8,%r9'
/tmp/cctOfDIJ.s:48768: Error: no such instruction: `shlx %rax,%r8,%r12'
/tmp/cctOfDIJ.s:48781: Error: no such instruction: `shlx %rax,%r8,%r12'
/tmp/cctOfDIJ.s:48795: Error: no such instruction: `shlx %rax,%r8,%rdi'
/tmp/cctOfDIJ.s:48806: Error: no such instruction: `shlx %rax,%r8,%rdi'
/tmp/cctOfDIJ.s:48822: Error: no such instruction: `shlx %rax,%r8,%r11'
/tmp/cctOfDIJ.s:48833: Error: no such instruction: `shlx %rax,%r8,%r11'
make: *** [.build_release/src/caffe/cpm/cpm_data_transformer.o] Error 1
/tmp/ccgfpXjY.s: Assembler messages:
/tmp/ccgfpXjY.s:13715: Error: no such instruction: `shlx %rsi,%r14,%r9'
/tmp/ccgfpXjY.s:13720: Error: no such instruction: `shlx %rdx,%r14,%rax'
/tmp/ccgfpXjY.s:13751: Error: no such instruction: `shlx %rbx,%rax,%rax'
/tmp/ccgfpXjY.s:13867: Error: no such instruction: `shlx %r10,%r9,%r14'
/tmp/ccgfpXjY.s:13873: Error: no such instruction: `shlx %rsi,%r9,%r10'
/tmp/ccgfpXjY.s:13875: Error: no such instruction: `shlx %rdx,%r9,%rax'
/tmp/ccgfpXjY.s:13929: Error: no such instruction: `shlx %rbx,%rdi,%r8'
/tmp/ccgfpXjY.s:13931: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/ccgfpXjY.s:14020: Error: no such instruction: `shlx %r8,%rax,%rax'
/tmp/ccgfpXjY.s:22921: Error: suffix or operands invalid for `vbroadcastss'
/tmp/ccgfpXjY.s:23068: Error: suffix or operands invalid for `vbroadcastss'
/tmp/ccgfpXjY.s:23187: Error: suffix or operands invalid for `vbroadcastss'
/tmp/ccgfpXjY.s:23363: Error: suffix or operands invalid for `vbroadcastss'
/tmp/ccgfpXjY.s:24987: Error: suffix or operands invalid for `vbroadcastsd'
/tmp/ccgfpXjY.s:25098: Error: suffix or operands invalid for `vbroadcastsd'
/tmp/ccgfpXjY.s:25189: Error: suffix or operands invalid for `vbroadcastsd'
/tmp/ccgfpXjY.s:25329: Error: suffix or operands invalid for `vbroadcastsd'
/tmp/ccgfpXjY.s:25564: Error: no such instruction: `shlx %rsi,%r13,%r11'
/tmp/ccgfpXjY.s:25569: Error: no such instruction: `shlx %rdx,%r13,%rax'
/tmp/ccgfpXjY.s:25706: Error: no such instruction: `shlx %rbx,%r8,%r9'
/tmp/ccgfpXjY.s:25724: Error: no such instruction: `shlx %rbx,%r8,%rax'
/tmp/ccgfpXjY.s:25748: Error: no such instruction: `shlx %rax,%r9,%r8'
/tmp/ccgfpXjY.s:25766: Error: no such instruction: `shlx %rax,%r9,%rdx'
/tmp/ccgfpXjY.s:25801: Error: no such instruction: `shlx %rbx,%r8,%r9'
/tmp/ccgfpXjY.s:25880: Error: no such instruction: `shlx %rsi,%r10,%r11'
/tmp/ccgfpXjY.s:25882: Error: no such instruction: `shlx %rax,%r10,%rdi'
/tmp/ccgfpXjY.s:26026: Error: no such instruction: `shlx %rax,%r8,%r11'
/tmp/ccgfpXjY.s:26039: Error: no such instruction: `shlx %rax,%r8,%r11'
/tmp/ccgfpXjY.s:26063: Error: no such instruction: `shlx %rbx,%r8,%rax'
/tmp/ccgfpXjY.s:26076: Error: no such instruction: `shlx %rax,%r9,%r8'
/tmp/ccgfpXjY.s:26086: Error: no such instruction: `shlx %rax,%r9,%rdx'
/tmp/ccgfpXjY.s:26137: Error: no such instruction: `shlx %rax,%r8,%rcx'
/tmp/ccgfpXjY.s:26163: Error: no such instruction: `shlx %rax,%rdi,%r12'
/tmp/ccgfpXjY.s:26175: Error: no such instruction: `shlx %rax,%r8,%rcx'
/tmp/ccgfpXjY.s:26188: Error: no such instruction: `shlx %rax,%rdi,%r12'
/tmp/ccgfpXjY.s:26205: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/ccgfpXjY.s:26249: Error: no such instruction: `shlx %rbx,%rdi,%r8'
/tmp/ccgfpXjY.s:26251: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/ccgfpXjY.s:26291: Error: no such instruction: `shlx %rax,%rdi,%rdx'
/tmp/ccgfpXjY.s:26322: Error: no such instruction: `shlx %rax,%r8,%r9'
/tmp/ccgfpXjY.s:26335: Error: no such instruction: `shlx %rax,%r8,%r9'
/tmp/ccgfpXjY.s:26363: Error: no such instruction: `shlx %rax,%r8,%r12'
/tmp/ccgfpXjY.s:26376: Error: no such instruction: `shlx %rax,%r8,%r12'
/tmp/ccgfpXjY.s:26390: Error: no such instruction: `shlx %rax,%r8,%rdi'
.
.
.

Is there any specific compiler/assembler instructions in code that I should look into? Such as forcing AVX/AVX2 optimizations?

In my testing, I found that for some image folders, rtpose.bin generate the exactly the same number of json files as that of image files in the source image folder. However, for some image folders, it doest not have the last two frame jsons. Is this behaviour as expected?

$ ls output_frame | tail
003496.jpg
003497.jpg
003498.jpg
003499.jpg
003500.jpg
003501.jpg
003502.jpg
003503.jpg
003504.jpg
003505.jpg
$ ls input_frame | tail
003498.jpg
003499.jpg
003500.jpg
003501.jpg
003502.jpg
003503.jpg
003504.jpg
003505.jpg
003506.jpg
003507.jpg

It is my test project: i just copy the file rtpose.cpp to the project. then i write a CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8)
PROJECT(test)
SET(CMAKE_CXX_COMPILER "g++")
SET(CMAKE_BUILD_TYPE Debug)
SET(CMAKE_CXX_FLAGS "-std=c++0x")
find_package(CUDA REQUIRED)
INCLUDE_DIRECTORIES(
${CUDA_INCLUDE_DIRS}
/home/wsh/projects/openpose/include
/home/wsh/projects/openpose/3rdparty/caffe/include
)
ADD_EXECUTABLE(pose rtpose.cpp)
TARGET_LINK_LIBRARIES(pose
${CUDA_LIBSS}
/home/wsh/projects/openpose/build/lib/libopenpose.so
/home/wsh/projects/openpose/3rdparty/caffe/build/lib/libcaffe.so
)
Then make error is :
/home/wsh/projects/openpose/include/openpose/experimental/face/faceExtractor.hpp:48:29: error: ‘ResizeAndMergeCaffe’ was not declared in this scope
std::shared_ptr<ResizeAndMergeCaffe> spResizeAndMergeCaffe;
/home/wsh/projects/openpose/include/openpose/experimental/face/faceExtractor.hpp:49:29: error: ‘NmsCaffe’ was not declared in this scope
std::shared_ptr<NmsCaffe> spNmsCaffe;
......
I can run the examples successfully in openpose, but in my test project, it can not. i do not understand it. is it a problem between makefile and cmake? How can I use openpose in another project?