ov-bert-demo.md

OpenVINO BERT/QA Demo

The following docker image is canned after following the demo guide here. https://docs.openvino.ai/latest/omz_demos_bert_question_answering_demo_python.html

launch docker runtime (detach mode)

container=vuiseng9/ov-ubt18_data_dev:2021.4.2-bert-pydemo

docker run -it \
    -d \
    --init \
    --privileged \
    -v ${HOME}:/hosthome \
    --shm-size 8G \
    ${container} bash

goto a terminal of the runtime above, run the following

cd /opt/intel/openvino_2021.4.752/deployment_tools/open_model_zoo/demos/bert_question_answering_demo/python/
./run_demo.sh

# ask some questions
who is bert?

Latency

cd /opt/intel/openvino_2021.4.752/deployment_tools/tools/benchmark_tool

python3 benchmark_app.py -b 1 -api sync -nireq 1 -niter 100 -m /opt/intel/openvino_2021.4.752/deployment_tools/open_model_zoo/demos/bert_question_answering_demo/python/intel/bert-large-uncased-whole-word-masking-squad-0001/FP32/bert-large-uncased-whole-word-masking-squad-0001.xml

python3 benchmark_app.py -b 1 -api sync -nireq 1 -niter 100 -m /opt/intel/openvino_2021.4.752/deployment_tools/open_model_zoo/demos/bert_question_answering_demo/python/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP32-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml

FPS

bapp=/opt/intel/openvino_2021.4.752/deployment_tools/tools/benchmark_tool/benchmark_app.py

python3 benchmark_app.py -b 1 -api async -niter 10000 -m $model

“-shape "result.1[1,128],result.2[1,128],result.3[1,128]"”

Throughput

bapp=/opt/intel/openvino_2021.4.752/deployment_tools/tools/benchmark_tool/benchmark_app.py

python3 $bapp -b 1 -api async -niter 1000 -m bert-base-squad-v1-fp32/bert-base-squad-v1-fp32.xml -shape "0[1,128],2[1,128],attention_mask[1,128]"
python3 $bapp -b 1 -api async -niter 1000 -m bert-base-squad-v1-8bit/bert-base-squad-v1-8bit.xml -shape "result.1[1,128],result.2[1,128],result.3[1,128]"
python3 $bapp -b 1 -api async -niter 1000 -m bert-squad-hybrid-filled-lt-fp32/bert-squad-hybrid-filled-lt-fp32.xml -shape "0[1,128],2[1,128],attention_mask[1,128]"
python3 $bapp -b 1 -api async -niter 1000 -m bert-squad-nncf-mvmt-filled-lt-fp32/bert-squad-nncf-mvmt-filled-lt-fp32.xml -shape "input.0[1,128],input.1[1,128],input.2[1,128]"
python3 $bapp -b 1 -api async -niter 1000 -m bert-squad-hybrid-filled-lt-8bit/bert-squad-hybrid-filled-lt-8bit.xml -shape "result.1[1,128],result.2[1,128],result.3[1,128]"
python3 $bapp -b 1 -api async -niter 1000 -m bert-squad-nncf-mvmt-filled-lt-8bit/bert-squad-nncf-mvmt-filled-lt-8bit.xml -shape "input.0[1,128],input.1[1,128],input.2[1,128]"

pip install openvino-dev==2022.1.0.dev20220302
or
https://test.pypi.org/project/openvino-dev/2022.1.0.dev20220224/