zhaoyanpeng

ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models

zhaoyanpeng

Preparing the Required Assets

ESPIRE uses two sources of 3D assets: commercial assets from TurboSquid and free assets from NVIDIA and other public sources. Below, we describe how to download and preprocess them. The pipeline relies on a binary file, runtime_payload.bin, which contains encrypted commercial assets and instructions for patching raw assets.

Due to licensing constraints, NVIDIA assets cannot be redistributed. Users must download them manually. To streamline this process, we provide a unified pipeline for acquiring all required assets.

To access the download script, run the following command and enter yes when prompted:

python patch.py --project 

zhaoyanpeng

Third-Party Asset Acquisition

A major portion of the 3D assets used in ESPIRE is sourced from NVIDIA Isaac Sim. Due to licensing constraints, these assets cannot be redistributed directly (see the asset details).

To facilitate installation, we provide two options for users to download the assets themselves. All downloaded assets should be placed under <ESPIRE_PROJ_ROOT>/data/assets.

Option 1 (Recommended: smaller and faster)

Ideally, users only download the assets required by ESPIRE. This can be done by running:

zhaoyanpeng

Aloha The Demonstrator & Controler

We aim to use only Aloha the Demonstrator to collect demonstrations, train a policy, and control the demonstrator itself. I will walk through the whole process using the official SDK, cobot_magic, provided by Agilex. This note largely follows the official tutorial of Agilex. Here I will focus on things customized for our goal.

cobot_magic

The SDK provides the following modules.

aloha-devel/   # policy learning and inference
camera_ws/     # camera configuration
collect_data/  # demonstration collection

zhaoyanpeng

Install ROS1-Noetic and ROS2-Humble from Source on Unubtu 22.04 Jammy

ATTENTION: Please use the default system-wide compiler tools like cmake, gcc, g++, and python, deactivate all conda/mamba environments, and remove all customized $PATH; otherwise, the compiler cmake may complain that necessary headers and libraries are missing, for example, "Could NOT find boost," "Compatibility with CMake < 3.5 has been removed from CMake," and "/usr/include/pybind11/detail/common.h:215:10: fatal error: Python.h: No such file or directory."

ROS2-Humble [REF]

There should not be any issues if you have been following the tutorial on a fresh OS, but there might be some pitfalls casued by your OS setups.

If it complains "Unable to locate package python3-rosdep" when installing development tools and ROS tools, simply change python3-rosdep to python3-rosdep2.

zhaoyanpeng

droot = "/home/xxxx/data"
kroot = f"{droot}/kelm-corpus/updated-2021"

kname = "kelm_generated_corpus.jsonl"
ename = "entities.jsonl"

tekgen_train = "quadruples-train.tsv"
tekgen_test = "quadruples-test.tsv"
tekgen_val = "quadruples-validation.tsv"

zhaoyanpeng

zhaoyanpeng

#!/usr/bin/sh

root=./

export OMP_NUM_THREADS=32
export CUDA_VISIBLE_DEVICES=$2

run_type=$1
gpu_list=(${CUDA_VISIBLE_DEVICES//,/ })

zhaoyanpeng

('three_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/three_hop.json')

How many <Z4> <C4> <M4> <S4>s are <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>?
What number of <Z4> <C4> <M4> <S4>s are <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>?
There is a <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; how many <Z4> <C4> <M4> <S4>s are <R3> it?
There is a <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; what number of <Z4> <C4> <M4> <S4>s are <R3> it?

There are <ANS> <Z4> <C4> <M4> <S4>s [that are] <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>.
The number of <Z4> <C4> <M4> <S4>s [that are] <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S> is <ANS>.
There is a <Z3> <C3>

zhaoyanpeng

('one_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/one_hop.json')

What number of <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>?
How many <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>?
There is a <Z> <C> <M> <S>; how many <Z2> <C2> <M2> <S2>s are <R> it?
There is a <Z> <C> <M> <S>; what number of <Z2> <C2> <M2> <S2>s are <R> it?

The number of <Z2> <C2> <M2> <S2>s [that are] <R> the <Z> <C> <M> <S> is <ANS>.
There are <ANS> <Z2> <C2> <M2> <S2>s [that are] <R> the <Z> <C> <M> <S>.
There is a <Z> <C> <M> <S>; there are <ANS> <Z2> <C2> <M2> <S2>s [that are] <R> it.