QAIC - Image Classification - Edge appliances

README.edge.md

Common parameters

Mandatory flags

Parameter	Description	Recorded in the name?	Recorded in the tags?
--model=resnet50	Workload name (only resnet50 for Image Classification)	yes (as resnet50)	yes (as workload.resnet50)
--sut=<string>	System-under-test	yes (as <string>)	yes (as platform.<string>)
--sdk=<string>	SDK version	yes (as qaic-<string>-aic100)	yes (as inference_engine_version.v<string>)
--scenario=<string>	LoadGen scenario	yes (as <string>)	yes (as scenario.<string>)
--mode=<string>	LoadGen mode	yes (as <string>)	yes (as mode.<string>)

Optional flags

Parameter	Description	Recorded in the name?	Recorded in the tags?
--dataset_size=<value>	Number of samples to process	yes (as dataset_size.<value>)	no (wrong?)
--buffer_size=<value>	Number of samples in buffer	no	yes (as buffer_size.<value>)
--timestamp	Timestamp the experiment	yes for --mode=performance no for --mode=accuracy	no

Offline

Accuracy

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=offline --mode=accuracy

krai@eb6:~/CK/local/experiment$ grep \"accuracy\": *resnet50*offline*accuracy*/*.0001.json -R
        "accuracy": 75.958,
          "accuracy": [

Optional parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_qps=<value> --offline_target_qps=<value> (alias)	LoadGen QPS (does not affect the speed or duration of the experiment)	no	no

Performance

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=offline --mode=performance --target_qps=1

krai@eb6:~/CK/local/experiment$ grep "Samples per second\"" *resnet50*offline*performance*/*.0001.json
          "Samples per second": "6510.15",

Mandatory parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_qps=<value> --offline_target_qps=<value> (alias)	LoadGen QPS (does not affect the speed of the experiment, only the duration)	yes (as target_qps.<value>)	no

Singlestream

Accuracy

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=singlestream --mode=accuracy

krai@eb6:~/CK/local/experiment$ grep \"accuracy\":\  *resnet50*singlestream*accuracy*/*.0001.json -R
        "accuracy": 75.946,
          "accuracy": [

Optional parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_latency=<value> --singlestream_target_latency=<value> (alias)	LoadGen Target Latency (does not affect the speed of the experiment, only the duration)	no	no

Performance

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=singlestream --mode=performance --target_latency=1000

krai@eb6:~/CK/local/experiment$ grep "90th percentile estimate\"" *resnet50*singlestream*/ckp*.0001.json -R | grep -v MLLOG
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-singlestream-performance-target_latency.1000/ckp-69d3de7586f34b6e.0001.json:          "* Early stopping 90th percentile estimate": "650501",

Mandatory parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_latency=<value> --singlestream_target_latency=<value> (alias)	LoadGen Target Latency (does not affect the speed of the experiment, only the duration)	yes (as target_latency.<value>)	no

Multistream

Accuracy

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=multistream --mode=accuracy 

krai@eb6:~/CK/local/experiment$ grep \"accuracy\":\  *resnet50*multistream*accuracy*/*.0001.json -R
        "accuracy": 75.946,
          "accuracy": [

Optional parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_latency=<value> --multistream_target_latency=<value> (alias)	LoadGen Target Latency (does not affect the speed of the experiment, only the duration)	no	no

Performance

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--scenario=multistream --mode=performance --target_latency=8000

krai@eb6:~/CK/local/experiment$ grep "99th percentile estimate\"" *resnet50*multistream*/ckp*.0001.json -R | grep -v MLLOG
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-multistream-performance-target_latency.8000/ckp-63ebb5a864e69932.0001.json:          "* Early stopping 99th percentile estimate": "2403679",

Mandatory parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--target_latency=<value> --multistream_target_latency=<value> (alias)	LoadGen Target Latency (does not affect the speed of the experiment, only the duration)	yes (as target_latency.<value>)	no

All-in-one

Accuracy

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--group.edge --mode=accuracy 

krai@eb6:~/CK/local/experiment$ grep \"accuracy\":\  *resnet50*accuracy*/*.0001.json -R
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-multistream-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-dbf8f804cfb4495f.0001.json:        "accuracy": 75.946,
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-multistream-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-dbf8f804cfb4495f.0001.json:          "accuracy": [
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-offline-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-f54bf9b45de7488a.0001.json:        "accuracy": 75.958,
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-offline-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-f54bf9b45de7488a.0001.json:          "accuracy": [
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-singlestream-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-7e3197ded7c4acfb.0001.json:        "accuracy": 75.946,
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-singlestream-accuracy-dataset_size.50000-preprocessed_using.opencv/ckp-7e3197ded7c4acfb.0001.json:          "accuracy": [

Mandatory parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--group.edge	runs three scenarios: --scenario=offline, --scenario=singlestream and --scenario=multistream	yes	yes (as scenario.<string>)

Performance

ck run cmdgen:benchmark.image-classification.qaic-loadgen --model=resnet50 --sut=eb6 --sdk=1.8.2 \
--group.edge --offline_target_qps=1 --singlestream_target_latency=1000 --multistream_target_latency=8000 \
--mode=performance

krai@eb6:~/CK/local/experiment$ grep "Samples per second\"" *resnet50*offline*performance*/*.0001.json
          "Samples per second": "6510.15",
krai@eb6:~/CK/local/experiment$ grep "90th percentile estimate\"" *resnet50*singlestream*/ckp*.0001.json -R | grep -v MLLOG
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-singlestream-performance-target_latency.1000/ckp-69d3de7586f34b6e.0001.json:          "* Early stopping 90th percentile estimate": "650501",
krai@eb6:~/CK/local/experiment$ grep "99th percentile estimate\"" *resnet50*multistream*/ckp*.0001.json -R | grep -v MLLOG
mlperf_v2.1-closed-eb6-qaic-v1.8.2-aic100-resnet50-multistream-performance-target_latency.8000/ckp-63ebb5a864e69932.0001.json:          "* Early stopping 99th percentile estimate": "2403679",

Mandatory parameters

Parameter	Description	Recorded in the name?	Recorded in the tags?
--group.edge	runs three scenarios: --scenario=offline, --scenario=singlestream and --scenario=multistream	yes	yes (as scenario.<string>)
--target_qps=<value> --offline_target_qps=<value> (alias)	LoadGen Target QPS (does not affect the speed of the experiment, only the duration)	yes (as target_qps.<value>)	no
--singlestream_target_latency=<value>	LoadGen Target Latency (does not affect the speed of the experiment, only the duration)	yes (as target_latency.<value>)	no
--multistream_target_latency=<value>	LoadGen Target latency (does not affect the speed of the experiment, only the duration)	yes (as target_latency.<value>)	no