Reusable Module for Operator tracking and gesture recognition

Modules are part of the ARISE Middleware

Important

Access the GitHub repository here!

Features

• Real-time hand detection using MediaPipe Hands

• 2D and 3D hand landmark estimation

• Dynamic multi-hand tracking pipeline

• Palm plane estimation and normal computation

• Camera-to-robot base frame transformation

• ROS2 Lifecycle Node architecture

• Runtime hand selection (left, right, both)

• Custom ROS2 interfaces and actions

• Built-in monitoring and diagnostics interface

Overview

This module provides a complete hand-tracking pipeline, including 2D hand detection, 3D landmark reconstruction, palm plane estimation and transformation into the robot reference frame.

Dependencies

All Python dependencies are included inside the requirements.txt file. To install, execute on terminal:

pip install -r /requirements.txt

This package is dependent on other ROS2 interfaces:

sudo apt install \
  ros-${ROS_DISTRO}-vision-msgs \
  ros-${ROS_DISTRO}-cv-bridge \
  ros-${ROS_DISTRO}-hri-msgs

What is included in the module

We provide the CARTIFactory package, home to several nodes, that together allow you to test this reusable module.

2D Hand detection

This node is the entry point of the hand-processing pipeline. It receives RGB images, performs hand landmark inference, identifies left and right hands, and publishes normalized 2D hand landmarks. The node also supports runtime hand selection (left, right, or both) through the orchestrator action interface.

Hand landmarks 3D node

The node synchronizes 2D hand detections with depth images and camera intrinsic parameters to estimate the 3D position of each MediaPipe landmark in the camera reference frame. Processing pipelines are created dynamically for every tracked hand.

Hand palm plane node

The node fits a plane using a configurable subset of hand landmarks and provides a stable estimation of palm orientation. This information can be used for gesture analysis, human-robot interaction, pointing estimation, or grasp intention inference.

Camera to robot frame hand transformation

Using the current robot TCP pose and a calibrated camera-to-TCP transformation, the node computes hand landmarks and palm planes expressed in robot coordinates. This enables downstream robotic applications to reason directly in the robot workspace.

Custom Interfaces

A custom_interfaces package is included, to handle the custom message for the pipeline information and the goal

Hand Tracking Message

Message definition for representing a tracked hand using MediaPipe landmarks:

HandDetection fields

Field	Type	Description
id	string	Unique identifier for the detected hand or tracked person.
handedness	uint8	Indicates whether the detected hand is left or right.
landmarks	hri_msgs/NormalizedPointOfInterest2D[21]	Array containing the 21 MediaPipe hand landmarks in normalized image coordinates.
confidence	float32	Confidence score of the hand detection/tracking.

Hand Tracking 3D Message

Message definition for representing a tracked hand in 3D space using MediaPipe landmarks.

HandDetection3D fields

Field	Type	Description
id	string	Unique identifier for the detected hand or tracked person.
handedness	uint8	Indicates whether the detected hand is left or right.
confidence	float32	Confidence score of the handedness classifier. If unavailable, use NaN or 0.
landmarks	geometry_msgs/Point[21]	Array containing the 21 MediaPipe hand landmarks in normalized image coordinates.
valid	bool[21]	Indicates whether each landmark is valid/tracked.

Lifecycle Architecture

At startup, all nodes remain in the unconfigured state. The module includes an orchestrator node that exposes ROS2 Actions for managing the lifecycle of the complete pipeline.

Supported lifecycle transitions:

• configure

• activate

• deactivate

• cleanup

• shutdown This architecture allows external applications to start, stop and reconfigure the entire hand-tracking pipeline without restarting the system.

Lifecycle action

All nodes will start as unconfigured upon launching the module. To unify the initialization of the system, we have created an orchestrator that hosts two actions, the first one being an action to change state to all the nodes in the correct order.

Goal

Goal fields

Field	Type	Description
target_state	string	State we want to change the system to.

Result

Result fields

Field	Type	Description
success	bool	Indicates whether the action execution completed successfully.
result_msg	string	Final message resulting from the execution of the action.

Feedback

Feedback fields

Field	Type	Description
feedback	string	Informational message describing the current status of the action execution.

Hand Change Action

The second action from the orchestrator is created for changing which hand is going to be tracked. Options are left, right or both.

Goal

Goal fields

Field	Type	Description
hand	string	Hand we want to track. Options: (left, right, both).

Result

Result fields

Field	Type	Description
success	bool	Indicates whether the action execution completed successfully.
result_msg	string	Final message resulting from the execution of the action.

Feedback

Feedback fields

Field	Type	Description
feedback	string	Informational message describing the current status of the action execution.

Running the Module

The launch file starts the full hand-tracking pipeline:

ros2 launch hand_hri_mediapipe hand_pipeline_lifecycle.launch.py

Launch arguments are the following:

Node parameters

Argument	Default	Type	Description
camera_name	cameraD455	string	Camera namespace used to build the color, depth, and camera info topics.
T_tcp_cam	identity/calibrated default	float[16]	Homogeneous transformation matrix from the robot TCP frame to the camera frame, used to transform hand landmarks and palm planes into the robot base frame.

Important

The T_tcp_cam matrix must be adapted to the real camera-to-robot calibration of the setup. An incorrect calibration will produce incorrect hand landmark and palm plane positions in the robot base frame.

---

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or HADEA. Neither the European Union nor the granting authority can be held responsible for them