graph TD;
A[This is an <b>important</b> <a href='https://gist.github.com/xihajun/fb7a7ece4d3ec9cd3f84671711853f87#a-host-setup'>link</a>] --> B[test is an <b>important</b> <a href='https://gist.github.com/xihajun/fb7a7ece4d3ec9cd3f84671711853f87#g-benchmarking'>test</a>]
generated by markdown-toc
Table of Contents
- Qualcomm Cloud AI - MLPerf Inference - Edge Appliances
- A. Host setup
- B. Initial device setup under the
rootuser - C. Initial device setup under the
kraiuser - D. Set up ImageNet
- E. Set up QAIC Platform SDK
- F. Install the workloads
- G. Benchmarking
- H. Inspecting the results
- Further info
We describe how to set up from scratch an Edge AI appliance similar to the Qualcomm Cloud AI 100 Edge Development Kit (AEDK), which we call "the device". We assume that the user operates an Ubuntu machine (or a Windows machine under WSL), which we call "the host".
Instructions below alternate between running on the host (marked with H)
and on the device (marked with D). Instructions to be run as superuser are
additionally marked with S.
Some instructions are to be run only once (marked with 1). Some instructions
are to be repeated as needed e.g. when updating to new SDK versions (marked with R).
We assume that the host machine has at least 50G of free space under a user-writable directory e.g. WORKSPACE_DIR=/workspace.
This space will be used to hold QAIC Apps/Platform SDKs, datasets (e.g. ImageNet), and Docker images.
export WORKSPACE_DIR=/workspace
mkdir -p $WORKSPACE_DIR/sdks
mkdir -p $WORKSPACE_DIR/datasets
Download the latest Apps SDK (e.g. AIC_Apps_SDK.ZIP.1.0 Installer x86_00058.1), x86_64/ubuntu Platform SDK (e.g. AIC_Platform_SDK.ZIP.1.0 Installer x86_Ubuntu_00069.1) and aarch64/ubuntu Platform SDK (e.g. AIC_Platform_SDK.ZIP.1.0 Installer ARM_Ubuntu_00069.1). Extract (unzip) the archives to obtain the Apps/Platform SDK archives of the same version (e.g. qaic-apps-sdk-1.8.2.10.zip, qaic-platform-sdk-x86_64-ubuntu-1.8.2.10.zip, qaic-platform-sdk-aarch64-ubuntu-1.8.2.10.zip). Copy these Apps/Platform SDK archives to $WORKSPACE_DIR/sdks.
Copy to $WORKSPACE_DIR/datasets the ImageNet dataset directory (called e.g. IMAGENET_NAME=imagenet), containing 50,000 images with the file names like ILSVRC2012_val_00000001.JPEG.
$ tree $WORKSPACE_DIR -L 2
/workspace
├── datasets
│ └── imagenet
└── sdks
├── qaic-apps-1.8.2.10.zip
├── qaic-platform-sdk-aarch64-ubuntu-1.8.2.10.zip
└── qaic-platform-sdk-x86_64-ubuntu-1.8.2.10.zip
3 directories, 3 files
git clone https://github.com/krai/ck-qaic /tmp/ck-qaic
cd /tmp/ck-qaic/script/setup.docker
WORKSPACE_DIR=/workspace bash setup_ck.sh
cd /tmp/ck-qaic/script/setup.docker
WORKSPACE_DIR=/workspace bash setup_ubuntu.sh
NB: Log out and log back in for the user to be added into the necessary groups.
WORKLOADS=resnet50 \
WORKSPACE_DIR=/workspace \
SDK_DIR=/workspace/sdks SDK_VER=1.8.2.10 \
DATASETS_DIR=/workspace/datasets IMAGENET_NAME=imagenet \
COMPILE_PRO=no COMPILE_STD=no \
DOCKER_OS=ubuntu bash setup_images.sh
If the host machine has at least QAIC PCIe card installed, you can test the image as follows:
ck run cmdgen:benchmark.image-classification.qaic-loadgen --docker --verbose \
--model=resnet50 --sdk=1.8.2.10 --sut=q1_std_edge --scenario=offline --mode=accuracy
...
accuracy=75.898%, good=37949, total=50000
Go to the following directory:
cd $(ck find repo:ck-qaic)/script/setup.aedk
and edit the config.sh file to update: the IP address (DEVICE_IP) and port (DEVICE_PORT) of the device, the path to the directory with datasets (HOST_DATASETS_DIR=$WORKSPACE/datasets).
NB: The full installation can take more than 50G. If the space on the root partition (/) of the device is limited and you wish to use a different partition to create the home directory under e.g. mounted as /data, then also update DEVICE_BASE_DIR=/data.
Init config.sh:
source $(ck find repo:ck-qaic)/script/setup.aedk/config.sh
Connect to the device as root e.g.:
ssh -p ${DEVICE_PORT} root@${DEVICE_IP}
git clone https://github.com/krai/ck-qaic /tmp/ck-qaic
Go to the temporary directory:
cd /tmp/ck-qaic/script/setup.aedk
Check the config file:
cat ./config.sh
#!/bin/bashexport DEVICE_IP=aedk3 export DEVICE_PORT=3233 export DEVICE_BASE_DIR="/home" export DEVICE_GROUP=krai export DEVICE_USER=krai export DEVICE_OS=ubuntu export DEVICE_OS_OVERRIDE=no export DEVICE_DATASETS_DIR="${DEVICE_BASE_DIR}/${DEVICE_USER}" export HOST_DATASETS_DIR="/datasets" export PYTHON_VERSION=3.9.14 export TIMEZONE="US/Central" export INSTALL_BENCHMARK_RESNET50=yes export INSTALL_BENCHMARK_BERT=no
Source it if you are happy with the settings and run:
source ./config.sh && ./1.run_as_root.sh
Alternatively, you can override variables from the command line e.g.:
time DEVICE_BASE_DIR=/data TIMEZONE=London/Europe ./1.run_as_root.sh
root@aus655-gloria-1:~# df -h /home Filesystem Size Used Avail Use% Mounted on /dev/root 99G 11G 89G 11% / root@aus655-gloria-1:~# df -h /datasets Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p1 880G 77M 835G 1% /datasets root@aus655-gloria-1:/tmp/ck-qaic/script/setup.aedk# time DEVICE_BASE_DIR=/datasets TIMEZONE=US/Central ./1.run_as_root.sh ... Sat Jul 23 09:05:56 CDT 2022 real 3m42.599s user 6m4.276s sys 1m5.008s
passwd ${DEVICE_USER}
Connect to the device as krai e.g.:
ssh -p ${DEVICE_PORT} krai@${DEVICE_IP}
sudo chown -R krai:krai /tmp/ck-qaic
sudo chmod u+x /tmp/ck-qaic/script/setup.aedk/*.sh
cd /tmp/ck-qaic/script/setup.aedk
source ./config.sh && time ./2.run_as_krai.sh
source ~/.bashrc
source ./config.sh && time ./3.run_as_krai.sh
source $(ck find repo:ck-qaic)/script/setup.aedk/config.sh
scp -r -P ${DEVICE_PORT} ${HOST_DATASETS_DIR}/imagenet krai@${DEVICE_IP}:${DEVICE_DATASETS_DIR}
echo "full" | ck detect soft:dataset.imagenet.val --extra_tags=ilsvrc2012,full \
--full_path=${DEVICE_DATASETS_DIR}/imagenet/ILSVRC2012_val_00000001.JPEG
ck install package --dep_add_tags.dataset-source=original,full \
--tags=dataset,imagenet,val,full,preprocessed,using-opencv,for.resnet50.quantized,layout.nhwc,side.224,validation
These steps are to be repeated for each new SDK version (SDK_VER below).
export SDK_VER=1.8.2.10
source $(ck find repo:ck-qaic)/script/setup.aedk/config.sh
scp -P ${DEVICE_PORT} \
${WORKSPACE}/sdks/qaic-platform-sdk-aarch64-ubuntu-${SDK_VER}.zip \
${DEVICE_USER}@${DEVICE_IP}:${DEVICE_BASE_DIR}/${DEVICE_USER}
Specify SDK_DIR, the path to a directory with one or more Platform SDK archives, and SDK_VER, the Platform SDK version.
The full path to the Platform SDK archive is formed as follows: PLATFORM_SDK=${SDK_DIR}/qaic-platform-sdk-aarch64-ubuntu-${SDK_VER}.zip.
SDK_DIR=${DEVICE_BASE_DIR}/${DEVICE_USER} SDK_VER=1.8.2.10 bash $(ck find ck-qaic:script:setup.aedk)/install_platform_sdk.sh
Alternatively, directly specify PLATFORM_SDK, the full path to the Platform SDK archive.
SDK_DIR=$HOME SDK_VER=1.8.2.10 $(ck find ck-qaic:script:setup.aedk)/install_platform_sdk.sh
Pretend that the device has the Apps SDK installed as well:
sudo cp /opt/qti-aic/versions/{platform,apps}.xml
Reboot the device for the changes to take effect.
cd $(ck find ck-qaic:script:setup.aedk)
WORKLOADS=resnet50 SDK_VER=1.8.2.10 \
DEVICE_IP=192.168.0.12 DEVICE_PORT=1234 DEVICE_PASSWORD=12345678 \
DEVICE_BASE_DIR=/home DEVICE_USER=krai DEVICE_TYPE=aedk ./install_to_aedk.sh
- On the host:
krai@aus655-el-01-5:/local/mnt/workspace/krai/CK-REPOS/ck-qaic/script/setup.aedk$ time WORKLOADS=resnet50 SDK_VER=1.8.2.10 DEVICE_USER=krai DEVICE_IP=192.168.0.12 DEVICE_PORT=1234 DEVICE_PASSWORD=12345678 DEVICE_BASE_DIR=/home DEVICE_TYPE=aedk ./install_to_aedk.sh ... DONE (installing workloads). real 6m58.061s user 0m3.306s sys 0m2.730s
- On the device:
krai@aus655-gloria-2:~$ ck show env --tags=qaic,model Env UID: Target OS: Bits: Name: Version: Tags: fd3d44a38096fd5c linux-64 64 Qualcomm Cloud AI model 1.8.2.12 64bits,bs.8,calibrated-by-qaic,compiled,compiled-by-qaic,converted,host-os-linux-64,image-classification,model,qaic,qualcomm,qualcomm-ai,qualcomm-cloud-ai,resnet50,resnet50.aedk.offline,target-os-linux-64,v1,v1.8,v1.8.2,v1.8.2.10,v1.8.2.12 43f4a098f915ab32 linux-64 64 Qualcomm Cloud AI model 1.8.2.12 64bits,bs.1,calibrated-by-qaic,compiled,compiled-by-qaic,converted,host-os-linux-64,image-classification,model,qaic,qualcomm,qualcomm-ai,qualcomm-cloud-ai,resnet50,resnet50.aedk.singlestream,target-os-linux-64,v1,v1.8,v1.8.2,v1.8.2.10,v1.8.2.12 ba6b84938baa22b5 linux-64 64 Qualcomm Cloud AI model 1.8.2.12 64bits,bs.1,calibrated-by-qaic,compiled,compiled-by-qaic,converted,host-os-linux-64,image-classification,model,qaic,qualcomm,qualcomm-ai,qualcomm-cloud-ai,resnet50,resnet50.aedk.multistream,target-os-linux-64,v1,v1.8,v1.8.2,v1.8.2.10,v1.8.2.12
ck run cmdgen:benchmark.image-classification.qaic-loadgen --verbose \
--model=resnet50 --sdk=1.8.2.10 --sut=eb6 --group.edge --group.open \
--offline_target_qps=1 --singlestream_target_latency=1000 \
--multistream_target_latency=1000 --multistream_query_count=660
krai@eb6:~$ ck list local:experiment:* | sort mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-multistream-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-multistream-performance-target_latency.1000 mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-offline-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-offline-performance-target_qps.1 mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-singlestream-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-singlestream-performance-target_latency.1000
ck run cmdgen:benchmark.image-classification.qaic-loadgen --verbose \
--model=resnet50 --sdk=1.8.2.10 --sut=eb6 --group.edge --group.open \
--offline_target_qps=1380 --singlestream_target_latency=5 \
--multistream_target_latency=10.5 --multistream_query_count=62857
krai@eb6:~$ ck list experiment:* | grep -v latency.1000 | sort mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-multistream-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-multistream-performance-target_latency.10.5 mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-offline-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-offline-performance-target_qps.1 mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-offline-performance-target_qps.1380 mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-singlestream-accuracy-dataset_size.50000-preprocessed_using.opencv mlperf_v2.1-closed-eb6-qaic-v1.8.2.10-aic100-resnet50-singlestream-performance-target_latency.5
ck run cmdgen:benchmark.image-classification.qaic-loadgen --verbose \
--model=resnet50 --sdk=1.8.2.10 --sut=eb6 --group.edge --group.open \
--offline_target_qps=6900 --singlestream_target_latency=1 \
--multistream_target_latency=2.1 --multistream_query_count=314285
Expand Details after individual Accuracy/Performance commands to see grep commands that can be used to quickly inspect the obtained results.
Please contact [email protected] if you have any problems or questions.