Simple Hand Mouvement Recognition Code - Hand tracking - Mediapipe

01-README.md

Simple Hand Mouvement Recognition Code - Hand tracking - Mediapipe

Goal of this gist is to recognize some simple hand mouvements like Scrolling, Zoom in/out and Slide left/right (see 08-hand-mouvement.gif below).

The whole code for that can be found here: hand-mouvement-recognition-calculator.cc.

You can clone my forked version of mediapipe here: https://github.com/TheJLifeX/mediapipe. I have already commited all code in that repository in the "hand-mouvement-recognition" branch (https://github.com/TheJLifeX/mediapipe/tree/hand-mouvement-recognition).

If you want to know how to recognize hand gesture like ONE, TWO, TREE, FOUR, FIVE, SIX, YEAH, ROCK, SPIDERMAN and OK. You can read this gist: Simple Hand Gesture Recognition.

Prerequisite: You kwon how to run the hand tracking example.

1. Get Started with mediapipe
2. Hand Tracking on Desktop

03-scrolling.md

Hand Mouvement: Scrolling

There are four defined scroll directions (up, right, down and left). To determine the scroll direction we have to calculate the angle between the current center of the hand closed (red) rectangle, the previous center and the x-axis.

• Scroll up: angle between 135° and 45°
• Scroll right: angle between 45° and -45°
• Scroll down: angle between -45° and -135°
• Scroll left: angle between -135° and 135°

Code for the hand mouvement "Scrolling" recogniton can be found here

04-scrolling.png

05-zoom-in-out.md

Hand Mouvement: Zoom in/out

The idea here is that when the hand closed (red) rectangle becomes bigger that will be recognized as "Zoom in", and when smaller as "Zoom out". I used only the height of the hand closed rectangle but the width or the height and width can be used.

Code for the hand mouvement "Zoom in/out" recogniton can be found here

06-slide-left-right.md

Hand Mouvement: Slide left/right

The idea here is that to slide left or the right the hand must be first upright. For that we define a interavall for the angle between the hand (wirst and MCP) and the x-axis. For the start position this angle is between 80° and 100°. We define this rule for a simple reason. After a "Slide right" for example the hand mouvement to the left should not be as "Slide left" recognized.

Code for the hand mouvement "Slide left/right" recogniton can be found here

07-slide-left-right.png

08-hand-mouvement.gif

09-hand-mouvement-recognition.mp4

@TheJLifeX,

I come across the below line in your code here,

float threshold = 0.01;
float closeFingerThreshold = 0.01;

float pseudoFixKeyPoint = landmarkList.landmark(2).x();
if (landmarkList.landmark(3).x() + threshold < pseudoFixKeyPoint && landmarkList.landmark(4).x() + threshold < landmarkList.landmark(3).x())
{
    thumbState = FingerState::OPEN;
}

else if (pseudoFixKeyPoint + threshold (<--SEE HERE) < landmarkList.landmark(3).x() && landmarkList.landmark(3).x() + threshold (<--SEE HERE) < landmarkList.landmark(4).x())
{
    thumbState = FingerState::CLOSE;
}

you used a threshold variable instead of "closeFingerThreshold " for both open and close cases.

is this behavior RIGHT?

please advise

Hi @SaddamBInSyed, it isn't available in python.

you used a threshold variable instead of "closeFingerThreshold " for both open and close cases.

It wasn't the right behavior. I have fixed it. Thank you!

Hi, i am not familiar with C++, that's hard for me to read the hand_Mouvement Recognition Code, Could u share a python version code for this code.

AnnotationOverlayCalculator: ; cc->Inputs().HasTag(recognizedHandMouvementScrollingTag)_overlay_calculator.cc:176)
Got this error while running the code. Any solution

your gesture code works well, but movement one, throwing this error

I0930 10:06:46.818581 5139 demo_run_graph_main.cc:53] Initialize the calculator graph.
E0930 10:06:46.820902 5139 demo_run_graph_main.cc:146] Failed to run the graph: ValidatedGraphConfig Initialization failed.
AnnotationOverlayCalculator: ; cc->Inputs().HasTag(recognizedHandMouvementScrollingTag)_overlay_calculator.cc:176)

running script: GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt