Use mediapipe to find multiple person 3d keypoints

mediapipe_multiple_person.py

import cv2
import numpy as np
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from mediapipe.framework.formats import landmark_pb2

# Model available to download here: https://developers.google.com/mediapipe/solutions/vision/pose_landmarker#models
model_path = "pose_landmarker_full.task"

video_source = 0

num_poses = 4
min_pose_detection_confidence = 0.5
min_pose_presence_confidence = 0.5
min_tracking_confidence = 0.5


def draw_landmarks_on_image(rgb_image, detection_result):
    pose_landmarks_list = detection_result.pose_landmarks
    annotated_image = np.copy(rgb_image)

    # Loop through the detected poses to visualize.
    for idx in range(len(pose_landmarks_list)):
        pose_landmarks = pose_landmarks_list[idx]

        pose_landmarks_proto = landmark_pb2.NormalizedLandmarkList()
        pose_landmarks_proto.landmark.extend([
            landmark_pb2.NormalizedLandmark(
                x=landmark.x,
                y=landmark.y,
                z=landmark.z) for landmark in pose_landmarks
        ])
        mp.solutions.drawing_utils.draw_landmarks(
            annotated_image,
            pose_landmarks_proto,
            mp.solutions.pose.POSE_CONNECTIONS,
            mp.solutions.drawing_styles.get_default_pose_landmarks_style())
    return annotated_image


to_window = None
last_timestamp_ms = 0


def print_result(detection_result: vision.PoseLandmarkerResult, output_image: mp.Image,
                 timestamp_ms: int):
    global to_window
    global last_timestamp_ms
    if timestamp_ms < last_timestamp_ms:
        return
    last_timestamp_ms = timestamp_ms
    # print("pose landmarker result: {}".format(detection_result))
    to_window = cv2.cvtColor(
        draw_landmarks_on_image(output_image.numpy_view(), detection_result), cv2.COLOR_RGB2BGR)


base_options = python.BaseOptions(model_asset_path=model_path)
options = vision.PoseLandmarkerOptions(
    base_options=base_options,
    running_mode=vision.RunningMode.LIVE_STREAM,
    num_poses=num_poses,
    min_pose_detection_confidence=min_pose_detection_confidence,
    min_pose_presence_confidence=min_pose_presence_confidence,
    min_tracking_confidence=min_tracking_confidence,
    output_segmentation_masks=False,
    result_callback=print_result
)

with vision.PoseLandmarker.create_from_options(options) as landmarker:
    # Use OpenCV’s VideoCapture to start capturing from the webcam.
    cap = cv2.VideoCapture(video_source)

    # Create a loop to read the latest frame from the camera using VideoCapture#read()
    while cap.isOpened():
        success, image = cap.read()
        if not success:
            print("Image capture failed.")
            break

        # Convert the frame received from OpenCV to a MediaPipe’s Image object.
        mp_image = mp.Image(
            image_format=mp.ImageFormat.SRGB,
            data=cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
        timestamp_ms = int(cv2.getTickCount() / cv2.getTickFrequency() * 1000)
        landmarker.detect_async(mp_image, timestamp_ms)

        if to_window is not None:
            cv2.imshow("MediaPipe Pose Landmark", to_window)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    cap.release()
    cv2.destroyAllWindows()

I find how to using this model with multiple person, Blaze Pose model from mediapipe pose model is operating only Pose Detect (two types of Modules from this model). then If we set all component of multiple pose(option num pose >= 2), 1st module(detector) were detect multi pose landmark at first video frame and 2nd module(Tracker) were using from frist frame pose landmark. so we can use multiple pose detection model

+) if you want to know detail of model. Enter my post link of multiple pose detetion model architecture.

Link : https://dablro12.tistory.com/12

In order to widely inform Koreans about this met hod, I wrote it in the Korean version. It's a hassle, but please understand.

did it work? I'm gonna convert it into mediapipe flugin unity

Maybe, I couldn't convert it into mediaipie plugin unity, so if you tried this methodm, Can you reply for me? [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality11&> 24. 04. 15. 오후 12:22:57 2024년 4월 15일 (월) 오후 12:15, Phạm Trần Triêm ***@***.***>님이 작성:

…

***@***.**** commented on this gist. ------------------------------ I find how to using this model with multiple person, Blaze Pose model from mediapipe pose model is operating only Pose Detect (two types of Modules from this model). then If we set all component of multiple pose(option num pose >= 2), 1st module(detector) were detect multi pose landmark at first video frame and 2nd module(Tracker) were using from frist frame pose landmark. so we can use multiple pose detection model [image: 다운로드] <https://private-user-images.githubusercontent.com/54443308/317915326-836b8684-a945-4615-8f99-0ec73c7498a7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTMxNTAwMDUsIm5iZiI6MTcxMzE0OTcwNSwicGF0aCI6Ii81NDQ0MzMwOC8zMTc5MTUzMjYtODM2Yjg2ODQtYTk0NS00NjE1LThmOTktMGVjNzNjNzQ5OGE3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDE1VDAyNTUwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRiYjlmZTg3MDdiMzNmMWU4MGRkNjk0ZmVmMGM4MTU3NDYwMWZkNWQ1NDhlYTgyNTg1ZTdhZTYzOGJkYWQ4MWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Ya2P1FyYToSWYrX2UgccqdMbJVd799iBiNXlIB058QQ> +) if you want to know detail of model. Enter my post link of multiple pose detetion model architecture. Link : https://dablro12.tistory.com/12 In order to widely inform Koreans about this met hod, I wrote it in the Korean version. It's a hassle, but please understand. did it work? I'm gonna convert it into mediapipe flugin unity — Reply to this email directly, view it on GitHub <https://gist.github.com/lanzani/f85175d8fbdafcabb7d480dd1bb769d9#gistcomment-5023346> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7L2LFKFKR2JDPH6SPTAP3Y5NA4JBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTENRUGM2DCNBUU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

-- [image: image.jpeg] 최대현 바이오메디컬공학 및 AI&Laguage m. 010-5661-6371 e. ***@***.*** p. choedaehyeon Portfolio <https://valiant-barnacle-6bf.notion.site/>

hey @lanzani ,
Thanks for sharing the script! I tried running it on my mac and when I have 2 people in camera frame, the skeletal landmarks keep switching between the 2 :(

based on what I read from this post and also the model info document , Pose Landmarker from mediapipe cannot detect multiple people in a single image. Even though you can set the num_poses parameter in options, the model itself only works when there is a single person present in the image.

just curious if the script worked on your end to output more than 1 person's skeletal landmarks?

Hi Federico,
Many thanks for posting this. Really helpful. Just to be on the safe side (as you do not specify a licence) - do I have your permission to re-use and extend this code in my own application? I will of course credit yourself as the originator. Would you also like me to clarify which parts of the code are yours and which I added (as per Apache 2.0)?
My extensions concern outputting landmark data to a .csv file and dealing with cases where two people are close together and only one is detected,

hey @lanzani , Thanks for sharing the script! I tried running it on my mac and when I have 2 people in camera frame, the skeletal landmarks keep switching between the 2 :(

I had the same problem with OpenPose. I ended up writing a postprocessor which addressed the switching by matching the keypoints for each person to the best fit in the previous frame, among other tweaks. I will share this when I have tidied it up!

based on what I read from this post and also the model info document , Pose Landmarker from mediapipe cannot detect multiple people in a single image. Even though you can set the num_poses parameter in options, the model itself only works when there is a single person present in the image.

just curious if the script worked on your end to output more than 1 person's skeletal landmarks?

i believe Mediapipe can detect multiple people in a single image but, as with video, only if they are far enough apart. I haven't quantified 'far enough' but it seems to be about the width of a person in the image.

hey @lanzani , Thanks for sharing the script! I tried running it on my mac and when I have 2 people in camera frame, the skeletal landmarks keep switching between the 2 :(

based on what I read from this post and also the model info document , Pose Landmarker from mediapipe cannot detect multiple people in a single image. Even though you can set the num_poses parameter in options, the model itself only works when there is a single person present in the image.

just curious if the script worked on your end to output more than 1 person's skeletal landmarks?

Hello! From what I remember, it worked. Keep in mind that I'm not developing with mediapipe since then, so probably something has changed :)

Hi Federico, Many thanks for posting this. Really helpful. Just to be on the safe side (as you do not specify a licence) - do I have your permission to re-use and extend this code in my own application? I will of course credit yourself as the originator. Would you also like me to clarify which parts of the code are yours and which I added (as per Apache 2.0)? My extensions concern outputting landmark data to a .csv file and dealing with cases where two people are close together and only one is detected,

Sure

hey @lanzani , Thanks for sharing the script! I tried running it on my mac and when I have 2 people in camera frame, the skeletal landmarks keep switching between the 2 :(

I had the same problem with OpenPose. I ended up writing a postprocessor which addressed the switching by matching the keypoints for each person to the best fit in the previous frame, among other tweaks. I will share this when I have tidied it up!

based on what I read from this post and also the model info document , Pose Landmarker from mediapipe cannot detect multiple people in a single image. Even though you can set the num_poses parameter in options, the model itself only works when there is a single person present in the image.
just curious if the script worked on your end to output more than 1 person's skeletal landmarks?

i believe Mediapipe can detect multiple people in a single image but, as with video, only if they are far enough apart. I haven't quantified 'far enough' but it seems to be about the width of a person in the image.

Unfortunately I'm not developing with mediapipe anymore, so I'm not much help :)

Hi Federico, Many thanks for posting this. Really helpful. Just to be on the safe side (as you do not specify a licence) - do I have your permission to re-use and extend this code in my own application? I will of course credit yourself as the originator. Would you also like me to clarify which parts of the code are yours and which I added (as per Apache 2.0)? My extensions concern outputting landmark data to a .csv file and dealing with cases where two people are close together and only one is detected,

Sure

Great, thanks :)