Notes on setting up ABB Follow Target Example on Isaac Sim 2022.2.0

follow_target_abb.md

Follow Target (ABB)

Notes on my ABB Follow Target Example extension for Omnniverse Isaac Sim 2022.2.0

Supports ABB IRB4600 and 8700

Set Up

Manual Instructions

To begin, you need to add and modify a few extensions in Isaac Sim 2022.2.0.

You can hop the installation folder through the Isaac Sim Settings in the OV Launcher.

Extension source directory is here: ~/.local/share/ov/pkg/isaac_sim-2022.2.0/exts

1. Copy follow_target_abb folder into omni.isaac.examples/omni/isaac/examples
2. In omni.isaac.examples/config/extension.toml:

• add "omni.isaac.abb" = {} under [dependencies]
• add "omni.osc.core" = {} and "omni.osc.server" = {} under [dependencies]
• add this to the end of the file:

[[python.module]]
name = "omni.isaac.examples.follow_target_abb"

3. Copy omni.isaac.abb, omni.osc.core, omni.osc.server into /isaac_sim-2022.2.0/exts.
4. In omni.isaac.motion_generation, add omni.isaac.motion_generation/motion_policy/abb to omni.isaac.motion_generation/motion_policy
5. In omni.isaac.motion_generation, add this as the last entry in omni.isaac.motion_generation/motion_policy/config.json

	"ABB_IRB4600":{
		"RMPflow" : "./abb/irb_4600/rmpflow/config.json"
	},
	"ABB_IRB8700":{
		"RMPflow" : "./abb/irb_8700/rmpflow/config.json"
	}

6. The last step is to manually fix a bug in omni.isaac.core.

Go to L141 in omni.isaac.core/omni/isaac/core/tasks/follow_target.py and change:

self._target.set_local_pose(position=target_position, orientation=target_orientation)

to

self._target.set_local_pose(translation=target_position, orientation=target_orientation)

Running the Follow Target Extension

1. Launch Isaac Sim from the OV Launcher or VSCode Terminal. I recommend running in VSCode:

• Open ~/.local/share/ov/pkg/isaac_sim-2022.2.0 VSCode folder.
• Terminal > New Terminal to open a new terminal.
• ./isaac_sim.sh to run Isaac Sim. This will allow hot reloading and hyperlinks to errors.

2. Go to the top toolbar and select Isaac Examples > Manipulation > Follow Target (ABB).
3. Press LOAD to add the robot to the scene.
4. Press START to begin following the target.
5. Select the TargetCube in the Viewport or through the Stage.
6. Press W / E when selected to Move / Rotate the TargetCube.

TO DO (Portland)

1. Calibrate Gestures
   • Motion Behavior Params are hard coded for much smaller robot
2. Calibrate Speed / Torque
   • Tune RMPFlow parameters to avoid high torque errors on robot in RobotStudio

   "Coupled Working Range" of Joints 2 & 3 and 5 & 6 cause Out of Coupled Range errors

3. Calibrate Obstacles
   • Match to meshes in RobotStudio

   RobotStudio Export Geometry not available in free version

Extra Features

Real-time Control

This extension can control physical ABB robots using an external EGM_Server app running on a separate Windows PC (so we can simulate in RobotStudio).

1. Connect to the Windows PC over ethernet and configure IPv4 to be on the same network as the Windows PC. For example:

hostname: 192.168.125.100
subnet: 255.255.255.0

2. Connect the Windows PC to the Robot Control Box, and configure it's network address to be on the same network as the Windows PC. For example:

hostname: 192.168.125.87
subnet: 255.255.255.0

3. Start the EGM_Server app on the Windows PC — there's also a desktop shortcut.
4. Press PP_to_Main and Play on the Teach Pendant to run the EGM_Main() task on the robot.

The robot should move to its [0, 0, 0, 0, 0, 0] position and TPWrite that it successfully connected to the EGM Server.

5. Connect Isaac Sim to the EGM_Server using the Quick Connect panel in Follow Target (ABB).

GUI Controls

• Address and Port should connect to the Windows PC running the EGM_Server.

RobotStudio Program

The .mod files for communicating with the real robot are in a directory called egm_template. Load the program file egm_template.pgf in RobotStudio and verify the following controller configurations:

IP Addresses for EGM:

Filters and Gains for EGM:

Motion Behaviors

GUI Controls

Receive Sensor Data

GUI Controls

Streaming Skeletons from Azure Kinect

You can stream the closest skeleton to Follow Target extension using an external openFrameworks app:

• Use my ofxAzureKinect addon for of_v0.11.2_linux64gcc6_release.
• I've modified the example-bodies app to broadcast the closest skeleton over localhost.

1. Run $OF_PATH/addons/ofxAzureKinect/example-bodies on the Linux PC.

#define HOST "localhost"
#define PORT 12345

2. In Isaac Sim, in the Follow Target > UDP Connection panel, press Connect to receive skeleton data.
3. Click Enable Tracking to connect the Motion Behaviors and Gestures to the incoming Skeleton.

Gestures

1. HANDS UP: Go Home
2. SKELETON ZONE IN ROBOT ZONE: Approach
3. HAND ZONE IN TOOL ZONE: Retract

NONE OF THESE GESTURES ARE CALIBRATED FOR THE IRB8700

Gamepad Control

Requires TouchOSC app and 8BitDo Lite2 controller

1. Connect the Gamepad to the iPad over Bluetooth.
2. Open TouchOSC and the Gamepad_Lite2.tosc file.
3. Press Play to verify TouchOSC is receiving Gamepad input.
4. In TouchOSC, configure the wifi setting to be on the same wifi network as the Linux PC running Isaac Sim.
5. In Isaac Sim, in the Follow Target > OSC Connection panel, add the Linux PC wifi address and press Connect.

The gamepad joysticks and X,Y,A,B buttons modify the TargetCube when not in MotionBehavior mode.

• LEFT_JOYSTICK: Moves target in XY
• RIGHT_JOYSTICK: Moves target in YZ
• A: Sends target to Reset Position
• X: Sends target to Home Position
• Y: Triggers Approach Mode

GUI Controls

OV OSC Extensions

In testing the Setup Instructions, I added the omni.osc.core and omni.osc.source source extensions, so I didn't need to add the deprecated OSC extenstions from the Extension Manager. Leaving these screenshots of my Extension Manager here for posterity: