Last active
September 16, 2025 20:57
-
-
Save AmosLewis/4c1ef357fafe07f7fb65fb41bac8d79f to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /opt/rocm/bin/rocprofv3 --output-format pftrace -r -- python3 -u harness_alt_mi355.py \ | |
| --devices "0,1,2,3,4,5,6,7" \ | |
| --scenario "$TEST_SCENARIO" \ | |
| --test_mode "$TEST_MODE" \ | |
| --bs 2 \ | |
| --user_conf_path user.conf \ | |
| --tensor_path /data/mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl \ | |
| --logfile_outdir "Output${TEST_SCENARIO}${TEST_MODE}" \ | |
| --debug "$DEBUG" \ | |
| --verbose "$VERBOSE" \ | |
| --user_conf_path "user.conf" \ | |
| --shortfin_config "$SHORTFIN_CONFIG" 2>&1 | tee server.log | |
| [0;33mW20250916 20:21:03.185427 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185455 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185456 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185458 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185459 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185460 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185461 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:03.185462 140463836505728 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [mINFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:03.583504 | |
| INFO:root:#################################################################################################################################################################################### | |
| Running python3 harness_alt_mi355.py --devices 0,1,2,3,4,5,6,7 --scenario Offline --test_mode PerformanceOnly --bs 2 --user_conf_path user.conf --count 8 --tensor_path /data/mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl --logfile_outdir OutputOfflinePerformanceOnly --debug False --verbose False --user_conf_path user.conf --shortfin_config shortfin_405b_config_fp4.json | |
| ############################################################################################################################################################################################## | |
| WARNING:root:Override count with 8 | |
| INFO:Llama-405B-Dataset:Loading dataset... | |
| INFO:Llama-405B-Dataset:Finished loading dataset. | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-2 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-3 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-4 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-5 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-6 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-7 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-8 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| Attempt to enable hip visiblity for agent-9 which is not visible to HSA (ROCR) | |
| [0;33mW20250916 20:21:06.222623 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222645 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222647 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222648 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222649 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222650 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222651 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222652 140286077806208 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222898 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222912 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222913 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222915 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222916 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222917 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222918 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.222919 140442072987264 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225667 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225692 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225693 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225695 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225696 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225697 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225698 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.225699 140074240882304 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230525 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230554 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230555 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230557 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230558 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230559 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230560 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.230561 140664284870272 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233050 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233073 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233074 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233075 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233076 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233077 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233078 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.233079 139864595117696 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236642 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236669 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236671 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236673 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236675 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236676 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236677 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.236678 139982489769600 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238224 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238253 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238255 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238256 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238257 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238258 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238259 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.238261 140159518242432 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239622 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239647 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239649 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239650 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239651 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239652 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239654 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.239655 139972769089152 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241451 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 2 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241476 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 3 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241478 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 4 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241480 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 5 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241481 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 6 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241482 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 7 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241483 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 8 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [m[0;33mW20250916 20:21:06.241484 140034939598464 metadata.cpp:210] rocprofiler_iterate_agent_supported_counters returned ROCPROFILER_STATUS_ERROR_AGENT_ARCH_NOT_SUPPORTED for agent 9 (gfx950) :: Agent HW architecture is not supported, no counter metrics found. | |
| [mINFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.641957 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.653119 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.656863 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.668896 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.674160 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.686915 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.688956 | |
| INFO:shortfin_apps.llm.components.service_debug_dumper:[debug_service.py] Please find debug dumps for service.py in /root/.shortfin/debug/llm_service_invocation_dumps/2025-09-16T20:21:06.705475 | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [0] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [1] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [0] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [1] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [1] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [1] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [0] | |
| INFO:root:NUMA hardware info: {'numa_node_distance': [[10, 32], [32, 10]], 'node_cpu_info': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191], 1: [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255]}} | |
| INFO:root:GPU: 0 | |
| INFO:root:Nearest nodes = [0] | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:shortfin_apps.llm.components.manager:Created local system with ['amdgpu:0:0@0'] devices | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Allocating page table (shape=[2816, 8257536], dtype=float8_e4m3fn, size=21.7GiB) on DeviceAffinity(amdgpu:0:0@0[0x1]) | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:root:Loading parameter fiber 'model' from: /shark-dev/weights/fp4/fp4_2025_07_10_fn.irpa | |
| INFO:shortfin_apps.llm.components.manager:Starting system manager | |
| INFO:root:Start Test! | |
| INFO:micro_llama_process_samples:SampleResponder-1 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-1 end time: 356602.527583834 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-1 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-1 end time: 356633.123653254 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-1 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-1 end time: 356656.755544615 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-1 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-1 end time: 356658.092370622 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-2 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-2 end time: 356690.782067668 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-2 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-2 end time: 356693.403412987 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-2 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-2 end time: 356764.89842253 | |
| ALLCLOSE: True | |
| INFO:micro_llama_process_samples:SampleResponder-1 Sending response | |
| INFO:micro_llama_process_samples:SampleResponder-1 end time: 357177.570724121 | |
| ALLCLOSE: True | |
| ================================================ | |
| MLPerf Results Summary | |
| ================================================ | |
| SUT name : PySUT | |
| Scenario : Offline | |
| Mode : PerformanceOnly | |
| Samples per second: 0.0118248 | |
| Tokens per second: 8.73556 | |
| Result is : VALID | |
| Min duration satisfied : Yes | |
| Min queries satisfied : Yes | |
| Early stopping satisfied: Yes | |
| ================================================ | |
| Additional Stats | |
| ================================================ | |
| Min latency (ns) : 101501116648 | |
| Max latency (ns) : 676544872396 | |
| Mean latency (ns) : 233618115916 | |
| 50.00 percentile latency (ns) : 189755781279 | |
| 90.00 percentile latency (ns) : 676544872396 | |
| 95.00 percentile latency (ns) : 676544872396 | |
| 97.00 percentile latency (ns) : 676544872396 | |
| 99.00 percentile latency (ns) : 676544872396 | |
| 99.90 percentile latency (ns) : 676544872396 | |
| ================================================ | |
| Test Parameters Used | |
| ================================================ | |
| samples_per_query : 8 | |
| target_qps : 1 | |
| ttft_latency (ns): 6000000000 | |
| tpot_latency (ns): 175000000 | |
| max_async_queries : 1 | |
| min_duration (ms): 6000 | |
| max_duration (ms): 0 | |
| min_query_count : 1 | |
| max_query_count : 8 | |
| qsl_rng_seed : 1780908523862526354 | |
| sample_index_rng_seed : 14771362308971278857 | |
| schedule_rng_seed : 18209322760996052031 | |
| accuracy_log_rng_seed : 0 | |
| accuracy_log_probability : 0 | |
| accuracy_log_sampling_target : 0 | |
| print_timestamps : 0 | |
| performance_issue_unique : 0 | |
| performance_issue_same : 0 | |
| performance_issue_same_index : 0 | |
| performance_sample_count : 8313 | |
| WARNING: sample_concatenate_permutation was set to true. | |
| Generated samples per query might be different as the one in the setting. | |
| Check the generated_samples_per_query line in the detailed log for the real | |
| samples_per_query value | |
| No warnings encountered during test. | |
| No errors encountered during test. | |
| INFO:root:[Server] Received 8 total samples | |
| [0;33mW20250916 20:33:28.168133 139972769089152 tool.cpp:1902] rocprofv3_error_signal_handler caught signal 15... | |
| [m[207.665] perfetto.cc:47304 Configured tracing session 1, #sources:1, duration:0 ms, #buffers:1, total buffer size:1048576 KB, total sessions:1, uid:0 session name: "" | |
| [0;31mE20250916 20:36:23.322229 139972769089152 output_stream.cpp:105] Opened result file: /mlperf/harness/smci350-zts-gtu-c8-25/8390_results.pftrace | |
| [m[405.366] perfetto.cc:48888 Tracing session 1 ended, total sessions:0 | |
| [0;33mW20250916 20:37:16.168115 140442072987264 tool.cpp:1902] rocprofv3_error_signal_handler caught signal 15... | |
| [m[435.668] perfetto.cc:47304 Configured tracing session 1, #sources:1, duration:0 ms, #buffers:1, total buffer size:1048576 KB, total sessions:1, uid:0 session name: "" | |
| ... | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment