Testing realsense-ros with a physical camera

README.md

testing realsense-ros

The pick and place simulation repo and the segmentation repo rely on camera functionality. We'd like to test rickstaa/realsense-ros-gazebo on a physical camera.

quickstart

Questions to answer:

• What is the dimensions of the color and depth channels on the camera?
• Does the ros module work on the physical camera?
• What is the behavior of the aligned depth image?

Ensure the realsense camera is plugged in and functional.

1. verify that the camera is working using the realsense viewer

Install and test the realsense2 ros module and test the gazebo module

1. checkout https://github.com/rickstaa/realsense-ros-gazebo into your catkin workspace under src
2. install dependencies via rosdep install --from-paths src --ignore-src -r -y
3. run catkin build
4. launch the gazebo example in the repo: roslaunch realsense2_description view_d435_model_rviz_gazebo.launch
5. save information from the simulated camera
   • use rostopic list and note all of the camera topics
   • use rostopic echo $TOPIC_NAME | head -n100 and get camera intrinsics information for the color and depth sensors

Run and test the physical camera via the ros module

1. read https://github.com/rickstaa/realsense-ros/tree/fa319702d34f1ca71149cddf5768bcb8e644f63b#usage-instructions
2. run roslaunch realsense2_camera rs_camera.launch
   • see instructions above and capture information about launched topics
3. configure the camera stream in rviz: http://wiki.ros.org/rviz
   • rosrun rviz rviz

Lastly, run the physical camera via ros module again, but set align_depth to true. Find out how to set parameters via roslaunch. Note down topics and camera intrinsics again for color and depth images.

Camera topics:

vipteam@bluestreak:~/azheng_ws$ rostopic list
/camera/color/camera_info
/camera/color/image_raw
/camera/color/image_raw/compressed
/camera/color/image_raw/compressed/parameter_descriptions
/camera/color/image_raw/compressed/parameter_updates
/camera/color/image_raw/compressedDepth
/camera/color/image_raw/compressedDepth/parameter_descriptions
/camera/color/image_raw/compressedDepth/parameter_updates
/camera/color/image_raw/theora
/camera/color/image_raw/theora/parameter_descriptions
/camera/color/image_raw/theora/parameter_updates
/camera/depth/camera_info
/camera/depth/color/points
/camera/depth/image_raw
/camera/depth/image_raw/compressed
/camera/depth/image_raw/compressed/parameter_descriptions
/camera/depth/image_raw/compressed/parameter_updates
/camera/depth/image_raw/compressedDepth
/camera/depth/image_raw/compressedDepth/parameter_descriptions
/camera/depth/image_raw/compressedDepth/parameter_updates
/camera/depth/image_raw/theora
/camera/depth/image_raw/theora/parameter_descriptions
/camera/depth/image_raw/theora/parameter_updates
/camera/infra1/camera_info
/camera/infra1/image_raw
/camera/infra1/image_raw/compressed
/camera/infra1/image_raw/compressed/parameter_descriptions
/camera/infra1/image_raw/compressed/parameter_updates
/camera/infra1/image_raw/compressedDepth
/camera/infra1/image_raw/compressedDepth/parameter_descriptions
/camera/infra1/image_raw/compressedDepth/parameter_updates
/camera/infra1/image_raw/theora
/camera/infra1/image_raw/theora/parameter_descriptions
/camera/infra1/image_raw/theora/parameter_updates
/camera/infra2/camera_info
/camera/infra2/image_raw
/camera/infra2/image_raw/compressed
/camera/infra2/image_raw/compressed/parameter_descriptions
/camera/infra2/image_raw/compressed/parameter_updates
/camera/infra2/image_raw/compressedDepth
/camera/infra2/image_raw/compressedDepth/parameter_descriptions
/camera/infra2/image_raw/compressedDepth/parameter_updates
/camera/infra2/image_raw/theora
/camera/infra2/image_raw/theora/parameter_descriptions
/camera/infra2/image_raw/theora/parameter_updates
/clicked_point
/clock
/diagnostics
/finalarm_joint_trajectory_action_controller/command
/finalarm_joint_trajectory_action_controller/follow_joint_trajectory/cancel
/finalarm_joint_trajectory_action_controller/follow_joint_trajectory/feedback
/finalarm_joint_trajectory_action_controller/follow_joint_trajectory/goal
/finalarm_joint_trajectory_action_controller/follow_joint_trajectory/result
/finalarm_joint_trajectory_action_controller/follow_joint_trajectory/status
/finalarm_joint_trajectory_action_controller/state
/finalarm_position_controller_1/command
/finalarm_position_controller_1/state
/finalarm_position_controller_2/command
/finalarm_position_controller_2/state
/finalarm_position_controller_3/command
/finalarm_position_controller_3/state
/finalarm_position_controller_4/command
/finalarm_position_controller_4/state
/finalarm_position_controller_5/command
/finalarm_position_controller_5/state
/finalarm_position_controller_6/command
/finalarm_position_controller_6/state
/finalarm_position_controller_7/command
/finalarm_position_controller_7/state
/finalarm_position_controller_8/command
/finalarm_position_controller_8/state
/finalarm_position_controller_9/command
/finalarm_position_controller_9/state
/gazebo/link_states
/gazebo/model_states
/gazebo/parameter_descriptions
/gazebo/parameter_updates
/gazebo/performance_metrics
/gazebo/set_link_state
/gazebo/set_model_state
/initialpose
/joint_states
/motor_states/port_0
/move_base_simple/goal
/rosout
/rosout_agg
/tf
/tf_static

Information from simulated camera (infra1):

rostopic echo /camera/infra1/camera_info | head -n100
header: 
  seq: 3073
  stamp: 
    secs: 389
    nsecs: 374000000
  frame_id: "camera_left_ir_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: []
K: [695.9951171875, 0.0, 640.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [695.9951171875, 0.0, 640.0, 0.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3074
  stamp: 
    secs: 389
    nsecs: 494000000
  frame_id: "camera_left_ir_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: []
K: [695.9951171875, 0.0, 640.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [695.9951171875, 0.0, 640.0, 0.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3075
  stamp: 
    secs: 389
    nsecs: 618000000
  frame_id: "camera_left_ir_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: []
K: [695.9951171875, 0.0, 640.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [695.9951171875, 0.0, 640.0, 0.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3076
  stamp: 
    secs: 389
    nsecs: 740000000
  frame_id: "camera_left_ir_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: []
K: [695.9951171875, 0.0, 640.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [695.9951171875, 0.0, 640.0, 0.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3077
  stamp: 
    secs: 389
    nsecs: 864000000
  frame_id: "camera_left_ir_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: []
K: [695.9951171875, 0.0, 640.0, 0.0, 695.9951171875, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

Information from simulated camera (color):

vipteam@bluestreak:~/azheng_ws$ rostopic echo /camera/color/camera_info | head -n100
header: 
  seq: 3275
  stamp: 
    secs: 414
    nsecs: 761000000
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: []
K: [462.1379699707031, 0.0, 320.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [462.1379699707031, 0.0, 320.0, 0.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3276
  stamp: 
    secs: 414
    nsecs: 881000000
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: []
K: [462.1379699707031, 0.0, 320.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [462.1379699707031, 0.0, 320.0, 0.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3277
  stamp: 
    secs: 415
    nsecs:   4000000
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: []
K: [462.1379699707031, 0.0, 320.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [462.1379699707031, 0.0, 320.0, 0.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3278
  stamp: 
    secs: 415
    nsecs: 123000000
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: []
K: [462.1379699707031, 0.0, 320.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [462.1379699707031, 0.0, 320.0, 0.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
header: 
  seq: 3279
  stamp: 
    secs: 415
    nsecs: 248000000
  frame_id: "camera_color_optical_frame"
height: 480
width: 640
distortion_model: "plumb_bob"
D: []
K: [462.1379699707031, 0.0, 320.0, 0.0, 462.1379699707031, 240.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

The command roslaunch realsense2_description view_d435_model_rviz_gazebo.launch opens rviz:

And gazebo:

Open realsense by entering:
realsense-viewer

Running the physical camera with roslaunch realsense2_camera rs_camera.launch and then running rostopic echo /camera/depth/camera_info gives an ongoing stream of things that look like:

---
header: 
  seq: 1102
  stamp: 
    secs: 1681243046
    nsecs:  30952930
  frame_id: "camera_depth_optical_frame"
height: 480
width: 848
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [420.2537841796875, 0.0, 424.7379150390625, 0.0, 420.2537841796875, 244.26292419433594, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [420.2537841796875, 0.0, 424.7379150390625, 0.0, 0.0, 420.2537841796875, 244.26292419433594, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---

The sequence number goes up each time
We think that the | head -n100 gives the BrokenPipeError somehow? We're not too sure what it does but it's probably printing that information at every frame?

We think that the | head -n100 gives the BrokenPipeError somehow? We're not too sure what it does but it's probably printing that information at every frame?

Just to clear this up in writing, rostopic echo will write to STDOUT continuously until it receives a signal from the user to stop. head is a unix command that will read the first n lines of STDIN and then exit. The broken pipe comes from head killing rostopic echo while it was running. And you are right that this is printing out information with every frame; there are other topics that are associated with accelerometers and other sensors that may need some information about the camera intrinsics here.

---

Linking a report with consolidated information here: https://gist.github.com/azheng987/2e714dc5097eaf1b50c0e175240fe1cb

Here is also a link that has a dockerfile that could be useful for using with the realsense: https://gist.github.com/acmiyaguchi/a0efca3b3265247d45bb4b8408e4b815