tachionstrahl / signlanguagerecognition Goto Github PK

I'm using ubuntu 18.04 and can build it smoothly, however, when I run ./run_prediction.sh it shows in the caption only "Buffer: X", where X is a number. The terminal shows the error:

Coordinates size not equal 86. Actual size: 128

screenshot attached.

Thank you for the useful repo.

I'm running MediaPipe on my mac and even though I have Bazel 3.5, I was able to run the Hello World example.

When running ./build_prediction.sh I get the following error:

ERROR: /private/var/tmp/bazel[hiddenusername]/8ace2bb3b3c7a44b1439e632e4b7edde/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin_x86_64'

I suspect this is because I don't have Bazel 3.4, but any suggestion is helpful! Thank you.

Hello are you pallning to release a version to run the model on an android device ?
Thank you

Hello, thanks for the great work on this repo! I understand that the repository doesn't currently support CPU operations. I am getting this error when running the demo commands in the README:

I20201111 20:41:01.626775 480495040 demo_run_graph_main.cc:53] Initialize the calculator graph.
E20201111 20:41:01.632501 480495040 demo_run_graph_main.cc:146] Failed to run the graph: ValidatedGraphConfig Initialization failed.
SignLangPredictionCalculator::GetContract failed to validate:
For output streams ValidatePacketTypeSet failed:
Failed to get tag "" index 0
Tag "TEXT" index 0 was not expected.

I would like to get this working on my end, would you be able to let me know roughly what changes should be made and where to make it functional?

Thanks!

Thanks for the great repo.

I follow step by step and the mediapipe hand tracker sample works fine on both cpu and gpu but by running ./build_prediction.sh
I faced with the error below.
How could I solve this?

ERROR: /.cache/bazel/_bazel/c6c43a719d1d619ea377f6e06c7a462c/external/mediapipe/mediapipe/util/tflite/BUILD:89:11: C++ compilation of rule '@mediapipe//mediapipe/util/tflite:tflite_gpu_runner' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 117 argument(s) skipped)

Based on your work I am attempting to create a calculator to extract landmarks from faceMesh. I managed to get the calculator working but am struggling to create the calculator options. I am trying to figure out , in your sign_lang_prediction_pose_gpu.pbtxt you have defined "signlang.SignLangPredictionCalculatorOptions.ext" based on your .proto, Where have you defined the "signlang" base class from where the calculator option is being called?
Thanks!

Please help me to know the structure of pickle file
I have combined all the json file features in text file for hands

Hello and thanks for the repo.

It was quite hard to get it to work, especially since mediapipe discontinued support for bazel<3.7 but now I'm encountering another problem.

Description: When running the demo the counter gets to 60, then after a while the program closes without ever showing a prediction. The webcam windows opens correctly and shows video input.
Full log:
./run_prediction.sh
I20210331 23:14:38.518226 12906 demo_run_graph_main_gpu.cc:51] Get calculator graph config contents: # Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"

node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:file_path"
options: {
[mediapipe.ConstantSidePacketCalculatorOptions.ext]: {
packet { string_value: "models/sign_lang_label_map.txt" }
}
}
}

node {
calculator: "LocalFileContentsCalculator"
input_side_packet: "FILE_PATH:file_path"
output_side_packet: "CONTENTS:contents"
}

node {
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:multi_hand_landmarks"
input_stream_info: {
tag_index: "FINISHED"
back_edge: true
}
output_stream: "throttled_input_video"
}

node {
calculator: "DetectionsGPUSubgraph"
input_stream: "throttled_input_video"
output_stream: "output_detections"
output_stream: "multi_hand_landmarks"
output_stream: "detection_render_data"
output_stream: "multi_hand_rects_render_data"
output_stream: "multi_palm_rects_render_data"
output_stream: "render_data"
output_stream: "multi_hand_landmarks_render_data"
}

node {
calculator: "SignLangPredictionCalculator"
input_side_packet: "LABELS:contents"
input_stream: "DETECTIONS:output_detections"
input_stream: "NORM_LANDMARKS:multi_hand_landmarks"
output_stream: "prediction"
options {
[signlang.SignLangPredictionCalculatorOptions.ext] {
verbose: false
framesWindowSize: 60
thresholdFramesCount: 5
minFramesForInference: 20
use3D: false
useRelative: false
probabilityThreshold: 0.5
fluentPrediction: false
tfLiteModelPath: "models/sign_lang_recognition_2D.tflite"
}
}
}

node {
calculator: "SentenizerCalculator"
input_stream: "prediction"
output_stream: "sentence"
}

node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "SIZE:image_size"
}

node {
calculator: "TextToRenderDataCalculator"
input_stream: "TEXT:sentence"
input_stream: "SIZE:image_size"
output_stream: "RENDER_DATA:0:background_rect"
output_stream: "RENDER_DATA:1:text_render_data"
}

node {
calculator: "AnnotationOverlayCalculator"
input_stream: "IMAGE_GPU:throttled_input_video"
input_stream: "render_data"
input_stream: "background_rect"
input_stream: "text_render_data"
input_stream: "VECTOR:0:multi_hand_landmarks_render_data"
output_stream: "IMAGE_GPU:output_video"
}
I20210331 23:14:38.518961 12906 demo_run_graph_main_gpu.cc:57] Initialize the calculator graph.
I20210331 23:14:38.522275 12906 demo_run_graph_main_gpu.cc:61] Initialize the GPU.
I20210331 23:14:38.532456 12906 gl_context_egl.cc:158] Successfully initialized EGL. Major : 1 Minor: 4
I20210331 23:14:38.535378 12920 gl_context.cc:324] GL version: 3.1 (OpenGL ES 3.1 Mesa 20.0.8)
I20210331 23:14:38.535461 12906 demo_run_graph_main_gpu.cc:67] Initialize the camera or load the video.
I20210331 23:14:38.870782 12906 demo_run_graph_main_gpu.cc:88] Start running the calculator graph.
I20210331 23:14:38.874580 12906 demo_run_graph_main_gpu.cc:93] Start grabbing and processing frames.
INFO: Created TensorFlow Lite delegate for GPU.
I20210331 23:14:39.423686 12920 gl_context.cc:807] Found unchecked GL error: GL_INVALID_FRAMEBUFFER_OPERATION
W20210331 23:14:39.423763 12920 gl_context.cc:828] Ignoring unchecked GL error.
E20210331 23:14:51.853036 12913 sign_lang_prediction_calculator.cc:366] Coordinates size not equal 86. Actual size: 128
E20210331 23:14:51.936754 12909 sign_lang_prediction_calculator.cc:366] Coordinates size not equal 86. Actual size: 128
E20210331 23:14:52.019819 12908 sign_lang_prediction_calculator.cc:366] Coordinates size not equal 86. Actual size: 128
E20210331 23:14:52.103752 12910 sign_lang_prediction_calculator.cc:366] Coordinates size not equal 86. Actual size: 128
E20210331 23:14:52.186794 12915 sign_lang_prediction_calculator.cc:366] Coordinates size not equal 86. Actual size: 128
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
./run_prediction.sh: line 1: 12906 Aborted (core dumped) GLOG_logtostderr=1 bazel-bin/app/prediction_gpu --calculator_graph_config_file=graphs/sign_lang_prediction_gpu.pbtxt`

System info:
`Ubuntu 18.04 (kernel 4.15.0-140-generic )
Python 3.6.9
Mediapipe 3.4.1
Mesa 20.0.8
GLX version: 1.4
OpenGL:
Max core profile version: 4.5
Max compat profile version: 3.0
Max GLES1 profile version: 1.1
Max GLES[23] profile version: 3.1

GPU nvidia GT740m`

Hi, thank you for this repo!

When I try to build and run the program, it was able to build successfully, but an issue occurs when I run it. Here's the output error:

I20201021 17:38:34.979502 11882 demo_run_graph_main_gpu.cc:57] Initialize the calculator graph.
I20201021 17:38:34.984632 11882 demo_run_graph_main_gpu.cc:61] Initialize the GPU.
libEGL warning: DRI2: failed to authenticate
I20201021 17:38:35.311697 11882 gl_context_egl.cc:158] Successfully initialized EGL. Major : 1 Minor: 4
I20201021 17:38:35.343811 11885 gl_context.cc:324] GL version: 3.1 (OpenGL ES 3.1 Mesa 20.0.8)
I20201021 17:38:35.343952 11882 demo_run_graph_main_gpu.cc:67] Initialize the camera or load the video.
E20201021 17:38:35.547452 11882 demo_run_graph_main_gpu.cc:186] Failed to run the graph: ; capture.isOpened()(external/mediapipe/mediapipe/examples/desktop/demo_run_graph_main_gpu.cc:75)

Would you happen to have encountered this and know a solution for it? I'm running Ubuntu 18.02 and Bazel 3.4.1.

Thank you!

Hello I am unable to find the code for building the CSV files from Frames. Any help would be very much appreciated.
Thanks!!