Skip to content

Instantly share code, notes, and snippets.

@surajssd

surajssd/01-README.md

Created April 28, 2025 22:36
Show Gist options
  • Save surajssd/c7dbf9e9202f182a8b61ca5fd96774f0 to your computer and use it in GitHub Desktop.
Save surajssd/c7dbf9e9202f182a8b61ca5fd96774f0 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
01-README.md

Run the following command to start editing the dcgm exporter daemonset:

kubectl -n gpu-operator edit ds nvidia-dcgm-exporter

Now add the following lines to the container spec:

        command:
        - sh
        - -c
        - /usr/local/dcgm/dcgm-exporter-entrypoint.sh --dcgm-log-level DEBUG --enable-dcgm-log

Update the image to the latest version, change the image to:

        image: nvcr.io/nvidia/k8s/dcgm-exporter:4.2.0-4.1.0-ubuntu22.04
Raw
exporter-4.1.1-4.0.4.sh
This file has been truncated, but you can view the full file.
$ kubectl logs -n gpu-operator nvidia-dcgm-exporter-6zr57
2025/04/28 22:29:00 maxprocs: Leaving GOMAXPROCS=96: CPU quota undefined
2025/04/28 22:29:00 INFO Starting dcgm-exporter Version=4.1.1-4.0.4
2025/04/28 22:29:00 INFO Attempting to initialize DCGM.
2025/04/28 22:29:00 INFO Initialized base logger [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:5309] [{anonymous}::StartEmbeddedV2] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Not changing to a home directory - 'DCGM_HOME_DIR' is not defined in the environment. [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:5322] [{anonymous}::StartEmbeddedV2] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO version:4.1.1;arch:x86_64;buildtype:RelWithDebInfo;buildid:11087;builddate:2025-02-14;commit:3965d2e947bcea4c496759177222de6115bd58d0;branch:v4.1.1;buildplatform:Linux 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64;;crc:84de5921dcda2d8986924b6bcea05213 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:5326] [{anonymous}::StartEmbeddedV2] dcgm_level=INFO
2025/04/28 22:29:00 INFO Env variable __DCGM_XID_KMSG__ unset. Not loading [/builds/dcgm/dcgm/dcgmlib/src/DcgmKmsgReader.cpp:40] [ReadEnvXidAndUpdate] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO __DCGM_TEST_KMSG_FILENAME__ unset. Not loading [/builds/dcgm/dcgm/dcgmlib/src/DcgmKmsgReader.cpp:149] [ReadEnvKmsgFilenameAndUpdate] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set m_forceProfMetricsThroughGpm to 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:946] [DcgmCacheManager::DcgmCacheManager] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Parsed driver string is 57012406, IsR450OrNewer: 1, IsR520OrNewer: 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2715] [DcgmCacheManager::ReadAndCacheDriverVersions] dcgm_level=INFO
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff21d0f8 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff281fc8 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff2e6e98 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff34bd68 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff3b0c38 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff415b08 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff47a9d8 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlDevice 0x7fb4ff4df8a8 is arch 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1559] [DcgmCacheManager::HelperGetLiveChipArch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Detected 12 NVLinks for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1278] [DcgmCacheManager::InitializeNvLinkCount] dcgm_level=INFO
2025/04/28 22:29:00 INFO [CacheManager][MIG] nvmlDeviceGetMigMode result: (0) Success. CurrentMode: 0, PendingMode: 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:707] [DcgmCacheManager::InitializeGpuInstances] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 0001:00:00.0 with GPU ID 0 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 0002:00:00.0 with GPU ID 1 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 0003:00:00.0 with GPU ID 2 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 0004:00:00.0 with GPU ID 3 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 000b:00:00.0 with GPU ID 4 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 000c:00:00.0 with GPU ID 5 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 000d:00:00.0 with GPU ID 6 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added GPU 000e:00:00.0 with GPU ID 7 to the pciBusGpuIdMap [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1256] [DcgmCacheManager::MergeNewlyDetectedGpuList] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Allowlist NOT bypassed with env variable [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1033] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, arch 7 is on the allowlist. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1057] [DcgmCacheManager::IsGpuAllowlisted] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 0 excluded GPUs [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:1093] [DcgmCacheManager::ReadAndCacheGpuExclusionList] dcgm_level=INFO
2025/04/28 22:29:00 INFO gpuId 0, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 1, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 2, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 3, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 4, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 5, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 6, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 7, desiredEvents x264, m_currentEventMask x0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2558] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 0 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 1 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 2 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 3 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 4 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 5 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 6 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Set nvmlIndex 7 event mask to x264 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2634] [DcgmCacheManager::ManageDeviceEvents] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Created thread named "cache_mgr_event" ID 278271552 DcgmThread ptr 0x0x7fb50805df10 [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:115] [DcgmThread::Start] dcgm_level=INFO
2025/04/28 22:29:00 INFO Thread handle 278271552 running [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:300] [DcgmThread::RunInternal] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO DcgmCacheManagerEventThread started [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:14001] [DcgmCacheManagerEventThread::run] dcgm_level=INFO
2025/04/28 22:29:00 INFO Created thread named "" ID 4203980352 DcgmThread ptr 0x0x7fb508034b80 [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:115] [DcgmThread::Start] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 0 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 1 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 2 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 3 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 4 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 5 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 6 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 0, eg 1, eid 7 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO Added GroupId 0 name DCGM_ALL_SUPPORTED_GPUS for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Env variable __DCGM_FATAL_XIDS__ unset. Not loading [/builds/dcgm/dcgm/dcgmlib/src/DcgmKmsgReader.cpp:40] [ReadEnvXidAndUpdate] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Thread handle 4203980352 running [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:300] [DcgmThread::RunInternal] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Entering dcgmModuleIdToName(dcgmModuleId_t id, char const **name) (1, 0x7fb513fbbf60) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:857] [dcgmModuleIdToName] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:857] [dcgmModuleIdToName] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Initialized logging for module 1 [/builds/dcgm/dcgm/modules/DcgmModule.h:90] [DcgmModuleWithCoreProxy<moduleId>::DcgmModuleWithCoreProxy] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loading NVSDM [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:29] [DcgmNs::createSwitchManager] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Initializing NVSDM Manager [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvsdmManager.cpp:582] [DcgmNs::DcgmNvsdmManager::Init] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Could not load NVSDM [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvsdmManager.cpp:621] [DcgmNs::DcgmNvsdmManager::AttachToNvsdm] dcgm_level=ERROR
2025/04/28 22:29:00 INFO [[NvSwitch]] AttachToNvsdm() returned -25 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvsdmManager.cpp:587] [DcgmNs::DcgmNvsdmManager::Init] dcgm_level=ERROR
2025/04/28 22:29:00 INFO [[NvSwitch]] Loading NSCQ [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:37] [DcgmNs::createSwitchManager] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not attached to NVSDM [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvsdmManager.cpp:641] [DcgmNs::DcgmNvsdmManager::DetachFromNvsdm] dcgm_level=WARN
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded NSCQ [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:506] [DcgmNs::DcgmNscqManager::AttachToNscq] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Created NSCQ session [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:521] [DcgmNs::DcgmNscqManager::AttachToNscq] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Attaching to NvSwitches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:557] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 phys id 12 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 phys id 13 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 phys id 8 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 phys id 9 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 phys id 10 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 phys id 11 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:584] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 6 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:593] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 12. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 12 at index 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 13. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 13 at index 1 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 8. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 8 at index 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 9. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 9 at index 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 10. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 10 at index 4 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Not found: phys id 11. Adding new switch [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:607] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Added switch: phys id 11 at index 5 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:631] [DcgmNs::DcgmNscqManager::AttachNvSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading switch status for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:674] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076d80 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076cb0 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076a40 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076b10 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb50805ff30 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076be0 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 6 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:722] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 1 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 4 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 5 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading NvLink states for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:762] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 9 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 13 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 14 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 12 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 31 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 12 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 8 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 28 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 25 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 9 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 24 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 29 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 30 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 15 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 13 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 14 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 15 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 8 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 216 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:826] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 9 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 13 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 14 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 12 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 31 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 12 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 8 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 28 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 25 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 9 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 24 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 29 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 30 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 15 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 13 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 14 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 15 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 8 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Finished reading NvLink states for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:847] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading fatal errors for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:898] [DcgmNs::DcgmNscqManager::ReadNvSwitchFatalErrorsAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 0 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:956] [DcgmNs::DcgmNscqManager::ReadNvSwitchFatalErrorsAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Constructing NvSwitch Module [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:55] [DcgmNs::DcgmModuleNvSwitch::DcgmModuleNvSwitch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Created thread named "" ID 4188010048 DcgmThread ptr 0x0x7fb5080615e8 [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:115] [DcgmThread::Start] dcgm_level=INFO
2025/04/28 22:29:00 INFO Loaded module 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:1943] [DcgmHostEngineHandler::LoadModule] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] Thread handle 4188010048 running [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:300] [DcgmThread::RunInternal] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Rescanning devices states [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:462] [DcgmNs::DcgmModuleNvSwitch::RescanDevicesState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading switch status for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:674] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076d80 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076cb0 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076a40 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076b10 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb50805ff30 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received device 0x7fb508076be0 denylist false [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:713] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 6 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:722] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 1 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 4 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Loaded status for switch at index 5 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:740] [DcgmNs::DcgmNscqManager::ReadNvSwitchStatusAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading NvLink states for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:762] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 9 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 13 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 14 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 12 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 31 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 12 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 8 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 28 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 25 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 32 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 9 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 14 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 24 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 28 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 35 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 29 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 30 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 34 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 15 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 12 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 15 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 8 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 31 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 32 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 13 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 14 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 9 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 15 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 25 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 13 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 18 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 33 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 30 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 19 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076b10 linkID 16 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 20 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 26 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 22 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 17 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 27 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 29 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 1 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 23 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 0 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb50805ff30 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 2 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076be0 linkID 24 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 3 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 5 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 6 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076cb0 linkID 33 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 7 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 8 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076a40 linkID 4 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 10 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 21 state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Received field Id 0 device 0x7fb508076d80 linkID 11 state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:810] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 216 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:826] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 9 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 13 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 14 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 12 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 31 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 12 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 8 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 28 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 25 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 32 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 9 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 14 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 24 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 28 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 35 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 29 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 30 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 34 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 15 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 12 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 15 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 8 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 31 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 32 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 13 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 14 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 9 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 15 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 25 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 13 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 18 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 33 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 30 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 19 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 4 link 16 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 20 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 26 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 22 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 17 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 27 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 29 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 1 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 23 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 0 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 2 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 2 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 5 link 24 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 3 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 5 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 6 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 0 link 33 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 7 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 8 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 3 link 4 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 10 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 21 to state 0 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Updating state for index 1 link 11 to state 2 [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:855] [DcgmNs::DcgmNscqManager::UpdateLinkState] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Finished reading NvLink states for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:847] [DcgmNs::DcgmNscqManager::ReadLinkStatesAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Reading fatal errors for all switches [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:898] [DcgmNs::DcgmNscqManager::ReadNvSwitchFatalErrorsAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] Callback called 0 times [/builds/dcgm/dcgm/modules/nvswitch/DcgmNscqManager.cpp:956] [DcgmNs::DcgmNscqManager::ReadNvSwitchFatalErrorsAllSwitches] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 12 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 13 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 8 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 9 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 10 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO [[NvSwitch]] No fields to update [/builds/dcgm/dcgm/modules/nvswitch/DcgmNvSwitchManagerBase.cpp:345] [DcgmNs::DcgmNvSwitchManagerBase::UpdateFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddEntityToGroup groupId 1, eg 3, eid 11 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:00 INFO Added GroupId 1 name DCGM_ALL_SUPPORTED_NVSWITCHES for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added field group id 1, name DCGM_INTERNAL_30SEC, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 8 entities and 1 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000000 (eg 1, entityId 0, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000001 (eg 1, entityId 1, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000002 (eg 1, entityId 2, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000003 (eg 1, entityId 3, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000004 (eg 1, entityId 4, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000005 (eg 1, entityId 5, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000006 (eg 1, entityId 6, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1012c00000007 (eg 1, entityId 7, fieldId 300) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 300, minMonitorFreqUsec 30000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 300, mfu 30000000, msa 14400, mka 480, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Skipping waitForUpdate since cache thread hasn't run yet. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2509] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added field group id 2, name DCGM_INTERNAL_HOURLY, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 8 gpus and 6 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3992] [DcgmHostEngineHandler::WatchFieldGroupAllGpus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000000 (eg 1, entityId 0, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000000 (eg 1, entityId 0, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000000 (eg 1, entityId 0, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000000 (eg 1, entityId 0, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000000 (eg 1, entityId 0, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000000 (eg 1, entityId 0, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 0, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000001 (eg 1, entityId 1, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000001 (eg 1, entityId 1, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000001 (eg 1, entityId 1, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000001 (eg 1, entityId 1, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000001 (eg 1, entityId 1, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000001 (eg 1, entityId 1, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 1, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000002 (eg 1, entityId 2, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000002 (eg 1, entityId 2, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000002 (eg 1, entityId 2, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000002 (eg 1, entityId 2, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000002 (eg 1, entityId 2, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000002 (eg 1, entityId 2, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 2, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000003 (eg 1, entityId 3, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000003 (eg 1, entityId 3, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000003 (eg 1, entityId 3, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000003 (eg 1, entityId 3, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000003 (eg 1, entityId 3, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000003 (eg 1, entityId 3, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 3, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000004 (eg 1, entityId 4, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000004 (eg 1, entityId 4, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000004 (eg 1, entityId 4, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000004 (eg 1, entityId 4, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000004 (eg 1, entityId 4, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000004 (eg 1, entityId 4, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 4, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000005 (eg 1, entityId 5, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000005 (eg 1, entityId 5, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000005 (eg 1, entityId 5, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000005 (eg 1, entityId 5, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000005 (eg 1, entityId 5, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000005 (eg 1, entityId 5, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 5, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000006 (eg 1, entityId 6, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000006 (eg 1, entityId 6, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000006 (eg 1, entityId 6, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000006 (eg 1, entityId 6, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000006 (eg 1, entityId 6, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000006 (eg 1, entityId 6, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 6, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101f500000007 (eg 1, entityId 7, fieldId 501) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 501, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 501, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fd00000007 (eg 1, entityId 7, fieldId 509) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 509, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 509, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101fe00000007 (eg 1, entityId 7, fieldId 510) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 510, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 510, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x101ff00000007 (eg 1, entityId 7, fieldId 511) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 511, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 511, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020000000007 (eg 1, entityId 7, fieldId 512) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 512, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 512, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding WatchInfo on entityKey 0x1020100000007 (eg 1, entityId 7, fieldId 513) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Adding new watcher type 1, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO UpdateWatchFromWatchers gid 1, eid 7, fid 513, minMonitorFreqUsec 3600000000, minMaxAgeUsec 15840000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO AddFieldWatch eg 1, eid 7, fieldId 513, mfu 3600000000, msa 14400, mka 4, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Skipping waitForUpdate since cache thread hasn't run yet. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2509] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Added field group id 3, name DCGM_INTERNAL_JOB, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Created thread named "cache_mgr_main" ID 4179617344 DcgmThread ptr 0x0x7fb5080598d0 [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:115] [DcgmThread::Start] dcgm_level=INFO
2025/04/28 22:29:00 INFO Waited 0 usec for the cache manager thread to start. [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2384] [DcgmCacheManager::Start] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Thread handle 4179617344 running [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:300] [DcgmThread::RunInternal] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Cache manager update thread starting [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6219] [DcgmCacheManager::run] dcgm_level=INFO
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508293110, eg 1, eid 6, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 501, ts 1745879340757289, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295680, eg 1, eid 0, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 510, ts 1745879340757329, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508293380, eg 1, eid 6, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 511, ts 1745879340757346, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294f10, eg 1, eid 3, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292200, eg 1, eid 3, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 511, ts 1745879340757371, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082932b0, eg 1, eid 6, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 510, ts 1745879340757385, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082954e0, eg 1, eid 0, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 501, ts 1745879340757404, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292540, eg 1, eid 4, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 509, ts 1745879340757418, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508290f80, eg 1, eid 3, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 501, ts 1745879340757432, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508291e80, eg 1, eid 2, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 510, ts 1745879340757449, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294d70, eg 1, eid 1, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082955b0, eg 1, eid 0, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 509, ts 1745879340757473, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295330, eg 1, eid 7, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295260, eg 1, eid 6, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294c80, eg 1, eid 0, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292b80, eg 1, eid 5, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 511, ts 1745879340757515, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294e40, eg 1, eid 2, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295c30, eg 1, eid 1, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 511, ts 1745879340757540, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082959c0, eg 1, eid 1, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 501, ts 1745879340757557, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295f70, eg 1, eid 2, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 509, ts 1745879340757571, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295750, eg 1, eid 0, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 511, ts 1745879340757588, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292ab0, eg 1, eid 5, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 510, ts 1745879340757601, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295d00, eg 1, eid 1, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 512, ts 1745879340757618, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292770, eg 1, eid 4, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 512, ts 1745879340757635, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295b60, eg 1, eid 1, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 510, ts 1745879340757649, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295fe0, eg 1, eid 7, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 509, ts 1745879340757666, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294fe0, eg 1, eid 4, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508293040, eg 1, eid 5, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 513, ts 1745879340757692, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295ea0, eg 1, eid 2, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 501, ts 1745879340757711, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292e20, eg 1, eid 3, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 509, ts 1745879340757728, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508291ae0, eg 1, eid 2, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 512, ts 1745879340757741, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082950b0, eg 1, eid 5, fieldId 300 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295820, eg 1, eid 0, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 512, ts 1745879340757775, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082958f0, eg 1, eid 0, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 0, fieldId 513, ts 1745879340757792, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292c50, eg 1, eid 5, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 512, ts 1745879340757808, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295a90, eg 1, eid 1, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 509, ts 1745879340757827, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292910, eg 1, eid 5, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 501, ts 1745879340757843, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508295dd0, eg 1, eid 1, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 1, fieldId 513, ts 1745879340757856, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508296050, eg 1, eid 7, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 510, ts 1745879340757875, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292470, eg 1, eid 4, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 501, ts 1745879340757889, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508294210, eg 1, eid 2, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 511, ts 1745879340757905, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292840, eg 1, eid 4, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 513, ts 1745879340757929, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508291d00, eg 1, eid 2, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 2, fieldId 513, ts 1745879340757945, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082931e0, eg 1, eid 6, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 509, ts 1745879340757965, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292ef0, eg 1, eid 3, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 510, ts 1745879340757978, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508296750, eg 1, eid 7, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 512, ts 1745879340757991, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082922d0, eg 1, eid 3, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 512, ts 1745879340758007, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082935f0, eg 1, eid 7, fieldId 501 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 501, ts 1745879340758023, valueSize 300, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082923a0, eg 1, eid 3, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 3, fieldId 513, ts 1745879340758039, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508292610, eg 1, eid 4, fieldId 510 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 510, ts 1745879340758061, valueSize 64, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082926a0, eg 1, eid 4, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 4, fieldId 511, ts 1745879340758077, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082929e0, eg 1, eid 5, fieldId 509 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 5, fieldId 509, ts 1745879340758100, valueSize 4, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508293450, eg 1, eid 6, fieldId 512 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 512, ts 1745879340758113, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb508293520, eg 1, eid 6, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 6, fieldId 513, ts 1745879340758132, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082960c0, eg 1, eid 7, fieldId 511 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 511, ts 1745879340758148, valueSize 2048, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Preparing to update watchInfo 0x7fb5082967f0, eg 1, eid 7, fieldId 513 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Checking status for gpu 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity blob eg 1, eid 7, fieldId 513, ts 1745879340758171, valueSize 4096, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 0, fieldId 300, ts 1745879340758193, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 1, fieldId 300, ts 1745879340758220, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 2, fieldId 300, ts 1745879340758234, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 3, fieldId 300, ts 1745879340758247, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 4, fieldId 300, ts 1745879340758264, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 5, fieldId 300, ts 1745879340758277, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 6, fieldId 300, ts 1745879340758290, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Got 1 field value fields for gpuId 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5589] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO fieldId 1 got good value type 3, value 0X1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5993] [DcgmCacheManager::ActuallyUpdateGpuFieldValues] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Appended entity i64 eg 1, eid 7, fieldId 300, ts 1745879340758307, value1 1, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Initialized DCGM Fields module.
2025/04/28 22:29:00 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO DCGM successfully initialized!
2025/04/28 22:29:00 INFO Attempting to initialize NVML library.
2025/04/28 22:29:00 INFO dcgmStartEmbedded(): Embedded host engine started [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:5350] [{anonymous}::StartEmbeddedV2] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO NVML provider successfully initialized!
2025/04/28 22:29:00 INFO Entering dcgmProfGetSupportedMetricGroups(dcgmHandle_t pDcgmHandle, dcgmProfGetMetricGroups_t *metricGroups) (2147483647, 0xc000624000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:823] [dcgmProfGetSupportedMetricGroups] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO nvmlGpmQueryDeviceSupport returned isSupportedDevice 0 for nvmlDevice x0x7fb4ff21d0f8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGpmManager.cpp:403] [DcgmGpmManager::DoesNvmlDeviceSupportGpm] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO gpuId 0 was not a GPM GPU [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:2035] [DcgmModuleCore::ProcessProfGetMetricGroups] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Entering dcgmModuleIdToName(dcgmModuleId_t id, char const **name) (8, 0x7fb513fbccb0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:857] [dcgmModuleIdToName] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:857] [dcgmModuleIdToName] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[Profiling]] Initialized logging for module 8 [/builds/dcgm/dcgm/modules/DcgmModule.h:90] [DcgmModuleWithCoreProxy<moduleId>::DcgmModuleWithCoreProxy] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[Profiling]] __DCGM_PROF_NO_SKU_CHECK was NOT set. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:579] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ReadEnvironmentalVariables] dcgm_level=DEBUG
2025/04/28 22:29:00 INFO [[Profiling]] NVPW_InitializeTarget() was successful. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1362] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_DCGM_LoadDriver() was successful. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1363] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_InitializeHost() was successful. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1364] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_GetDeviceCount() was successful [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1377] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetPciBusIds() was successful. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1394] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 1) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 2) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 3) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 4) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 11) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 12) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 13) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful (device: 0, bus: 0, domain: 14) [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1416] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 63, ts 1745879341039591, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 0, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 0 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 0 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 0 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 0 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 0 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 0 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 0 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 0 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 0 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 0 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 0 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 0 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 0 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 0 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 0. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 63, ts 1745879341039911, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 1, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 1 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 1 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 1 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 1 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 1 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 1 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 1 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 1 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 1 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 1 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 1 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 1 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 1 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 1 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 1. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 2 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 2 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 63, ts 1745879341040216, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 2, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 2 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 2 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 2 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 2 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 2 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 2 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 2 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 2 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 2 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 2 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 2 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 2 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 2 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 2 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 2. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 63, ts 1745879341040499, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 3, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 3 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 3 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 3 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 3 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 3 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 3 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 3 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 3 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 3 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 3 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 3 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 3 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 3 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 3 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 3. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 63, ts 1745879341040774, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 4, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 4 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 4 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 4 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 4 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 4 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 4 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 4 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 4 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 4 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 4 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 4 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 4 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 4 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 4 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 4. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 63, ts 1745879341041054, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 5, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 5 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 5 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 5 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 5 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 5 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 5 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 5 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 5 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 5 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 5 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 5 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 5 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 5 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 5 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 5. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 6 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 6 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 63, ts 1745879341041325, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 6, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 6 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 6 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 6 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 6 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 6 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 6 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 6 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 6 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 6 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 6 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 6 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 6 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 6 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 6 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 6. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] NVPW_Device_GetMigAttributes was successful 7 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:68] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Init was successful for deviceIndex 7 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmLopGpu.cpp:86] [DcgmLopGpu::Init] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 7, fieldId 63, ts 1745879341041619, value1 524288, value2 1, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Building LOP metrics for GPU with CC 8.0 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1195] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id GrActive, lopTag gr__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id SmActive, lopTag sm__cycles_active.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id SmOccupancy, lopTag sm__warps_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeTensorActive, lopTag sm__pipe_tensor_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id DramActive, lopTag dram__throughput.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeFp64Active, lopTag sm__pipe_fp64_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeFp32Active, lopTag smsp__inst_executed_pipe_fma_op_ffma.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeFp16Active, lopTag sm__inst_executed_pipe_fp16_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PcieTxBytes, lopTag pcie__write_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PcieRxBytes, lopTag pcie__read_bytes.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id NvLinkTxBytes, lopTag nvltx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id NvLinkRxBytes, lopTag nvlrx__bytes_data_user.sum.per_second is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeTensorImmaActive, lopTag sm__pipe_tensor_op_imma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] gpuId 7, id PipeTensorHmmaActive, lopTag sm__pipe_tensor_op_hmma_cycles_active_realtime.avg.pct_of_peak_sustained_elapsed is supported. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1261] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildLopMetricList] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1001} is supported for gpuId 7 as metricIndex GrActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric GrActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1002} is supported for gpuId 7 as metricIndex SmActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1003} is supported for gpuId 7 as metricIndex SmOccupancy [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric SmOccupancy entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1004} is supported for gpuId 7 as metricIndex PipeTensorActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1005} is supported for gpuId 7 as metricIndex DramActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric DramActive entity level 4 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1006} is supported for gpuId 7 as metricIndex PipeFp64Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp64Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1007} is supported for gpuId 7 as metricIndex PipeFp32Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp32Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1008} is supported for gpuId 7 as metricIndex PipeFp16Active [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeFp16Active entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1009} is supported for gpuId 7 as metricIndex PcieTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1010} is supported for gpuId 7 as metricIndex PcieRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PcieRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1011} is supported for gpuId 7 as metricIndex NvLinkTxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkTxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1012} is supported for gpuId 7 as metricIndex NvLinkRxBytes [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric NvLinkRxBytes entity level 1 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1013} is supported for gpuId 7 as metricIndex PipeTensorImmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorImmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] fieldId FieldId {1014} is supported for gpuId 7 as metricIndex PipeTensorHmmaActive [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1162] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Assigning metric PipeTensorHmmaActive entity level 5 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1178] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::TryAddingFieldIdWithMetric] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] BuildMetricGroupsGA100 result 0b1111111 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:827] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::BuildMetricGroupsGA100] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] LOP monitoring initialized successfully for gpuId 7. [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1540] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::InitLop] dcgm_level=INFO
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 114, ts 1745879341041879, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 114, ts 1745879341044326, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 114, ts 1745879341046732, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 114, ts 1745879341049158, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 114, ts 1745879341051572, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 114, ts 1745879341053975, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 114, ts 1745879341056407, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 7, fieldId 114, ts 1745879341058824, value1 1215, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Got mem clock 1215 for gpuId 0. fv->status [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:924] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::UpdateMemoryClocksAndBw] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Debug logging is enabled for the TaskRunner at 0x7fb5082aa1e0 [/builds/dcgm/dcgm/common/TaskRunner.hpp:110] [DcgmNs::TaskRunner::SetDebugLogging] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Debug logging is enabled for the Semaphore at 0x7fb5082aa420 [/builds/dcgm/dcgm/common/Semaphore.hpp:380] [DcgmNs::Semaphore::SetDebugLogging] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Created thread named "" ID 3424880192 DcgmThread ptr 0x0x7fb5082aa148 [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:115] [DcgmThread::Start] dcgm_level=INFO
2025/04/28 22:29:01 INFO Loaded module 8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:1943] [DcgmHostEngineHandler::LoadModule] dcgm_level=INFO
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Thread handle 3424880192 running [/builds/dcgm/dcgm/common/DcgmThread/DcgmThread.cpp:300] [DcgmThread::RunInternal] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] minWakeupMs = 10000 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1739] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ReadAndCacheMetrics] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Call RunOnce after all events are processed' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Call RunOnce after all events are processed' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] minWakeupMs = 10000 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1739] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ReadAndCacheMetrics] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:823] [dcgmProfGetSupportedMetricGroups] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Call RunOnce after all events are processed' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Collecting DCP Metrics
2025/04/28 22:29:01 INFO Falling back to metric file '/etc/dcgm-exporter/dcp-metrics-included.csv'
2025/04/28 22:29:01 INFO Initializing system entities of type 'GPU'
2025/04/28 22:29:01 INFO Entering dcgmGetAllDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc00079c200 0xc0001dc1e8) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:81] [dcgmGetAllDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:81] [dcgmGetAllDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 0 0xc000236000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 158, ts 1745879341063194, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 159, ts 1745879341066386, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 164, ts 1745879341066566, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 160, ts 1745879341070456, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 163, ts 1745879341070736, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 162, ts 1745879341070743, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 161, ts 1745879341070753, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 130, ts 1745879341070758, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 54, ts 1745879341072036, value "GPU-5172a8c2-39f0-f1b4-f354-9fd17ec1f9b3", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 85, ts 1745879341072044, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 82, ts 1745879341072064, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 51, ts 1745879341072079, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 50, ts 1745879341072086, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 53, ts 1745879341072099, value "1321720104118", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 57, ts 1745879341072119, value "00000001:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 58, ts 1745879341072127, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 59, ts 1745879341072134, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 90, ts 1745879341072139, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 250, ts 1745879341072389, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 252, ts 1745879341072413, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 251, ts 1745879341072438, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 1, ts 1745879341072463, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 500, ts 1745879341072473, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 66, ts 1745879341072478, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 67, ts 1745879341082486, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 74, ts 1745879341082491, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc00079c280 0xc0001dc1ec) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc0001dc200, cpuAffFields12752557072830595745, 0xc0001dc220) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 4, name cpuAffFields12752557072830595745, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff9063882802458552645 0xc0001dc230) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 2 name cpuAff9063882802458552645 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 2 0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 2, eg 1, eid 0 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 2 added eg 1, eid 0. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 2, 4, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000000 (eg 1, entityId 0, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000000 (eg 1, entityId 0, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000000 (eg 1, entityId 0, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000000 (eg 1, entityId 0, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e23f0, eg 1, eid 0, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 73, ts 1745879341083411, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082b7a10, eg 1, eid 0, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 70, ts 1745879341358072, value1 281474959933440, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082adab0, eg 1, eid 0, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 72, ts 1745879341358092, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082ada40, eg 1, eid 0, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 71, ts 1745879341358109, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 0 0xc0001dc200 4 0xc0007aa000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 2) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 4) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 0 0xc00026c1c0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341359864, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341360040, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341371707, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 0 0xc000238a00) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 158, ts 1745879341372983, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 159, ts 1745879341373004, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 164, ts 1745879341373329, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 160, ts 1745879341373341, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 163, ts 1745879341373537, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 162, ts 1745879341373544, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 0, fieldId 161, ts 1745879341373550, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 130, ts 1745879341373556, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 54, ts 1745879341373571, value "GPU-5172a8c2-39f0-f1b4-f354-9fd17ec1f9b3", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 85, ts 1745879341373578, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 82, ts 1745879341373585, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 51, ts 1745879341373591, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 50, ts 1745879341373600, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 53, ts 1745879341373607, value "1321720104118", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 57, ts 1745879341373613, value "00000001:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 58, ts 1745879341373620, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 59, ts 1745879341373626, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 90, ts 1745879341373632, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 250, ts 1745879341373889, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 252, ts 1745879341373914, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 251, ts 1745879341373937, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 0, fieldId 1, ts 1745879341373961, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 500, ts 1745879341373971, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 66, ts 1745879341373976, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 67, ts 1745879341374174, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 74, ts 1745879341374178, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc0004fa4e8, pciBandwidthFields18106666993540717272, 0xc0004fa508) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 5, name pciBandwidthFields18106666993540717272, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth16224954995971507456 0xc0004fa518) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 3 name pciBandwidth16224954995971507456 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 3 0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 3, eg 1, eid 0 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 3 added eg 1, eid 0. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 3, 5, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000000 (eg 1, entityId 0, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000000 (eg 1, entityId 0, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 0, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 0, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e2af0, eg 1, eid 0, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 236, ts 1745879341374405, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e2b60, eg 1, eid 0, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 0, fieldId 235, ts 1745879341374532, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 0 0xc0004fa4e8 2 0xc00072c000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 5) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 3) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 1 0xc00033d500) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 158, ts 1745879341375780, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 159, ts 1745879341378966, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 164, ts 1745879341379148, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 160, ts 1745879341383082, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 163, ts 1745879341383357, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 162, ts 1745879341383363, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 161, ts 1745879341383368, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 1, fieldId 130, ts 1745879341383374, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 54, ts 1745879341384704, value "GPU-9e04693b-5415-708e-234e-92d2cf45e1c1", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 85, ts 1745879341384711, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 82, ts 1745879341384730, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 51, ts 1745879341384740, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 50, ts 1745879341384746, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 53, ts 1745879341384760, value "1322820084920", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 57, ts 1745879341384771, value "00000002:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 58, ts 1745879341384777, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 59, ts 1745879341384784, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 90, ts 1745879341384789, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 250, ts 1745879341385030, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 252, ts 1745879341385052, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 251, ts 1745879341385071, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 1, ts 1745879341385097, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 500, ts 1745879341385105, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 66, ts 1745879341385110, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 67, ts 1745879341385223, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 74, ts 1745879341385227, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc0006ac380 0xc0004fa560) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc0004fa568, cpuAffFields6095670426820071448, 0xc0004fa588) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 6, name cpuAffFields6095670426820071448, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff10608953186922503823 0xc0004fa598) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 4 name cpuAff10608953186922503823 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 4 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 4, eg 1, eid 1 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 4 added eg 1, eid 1. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 4, 6, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000001 (eg 1, entityId 1, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000001 (eg 1, entityId 1, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000001 (eg 1, entityId 1, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000001 (eg 1, entityId 1, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e3ca0, eg 1, eid 1, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 71, ts 1745879341385458, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e2bd0, eg 1, eid 1, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 70, ts 1745879341385491, value1 281474959933440, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e3d50, eg 1, eid 1, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 72, ts 1745879341385508, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5082e41c0, eg 1, eid 1, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 73, ts 1745879341385523, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 1 0xc0004fa568 4 0xc00073a000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 4) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 6) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 1 0xc0006b6000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341386136, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341386289, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341397768, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 1 0xc000342000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 158, ts 1745879341399090, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 159, ts 1745879341399117, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 164, ts 1745879341399397, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 160, ts 1745879341399413, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 163, ts 1745879341399625, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 162, ts 1745879341399631, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 1, fieldId 161, ts 1745879341399639, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 1, fieldId 130, ts 1745879341399645, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 54, ts 1745879341399689, value "GPU-9e04693b-5415-708e-234e-92d2cf45e1c1", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 85, ts 1745879341399697, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 82, ts 1745879341399705, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 51, ts 1745879341399711, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 50, ts 1745879341399717, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 53, ts 1745879341399727, value "1322820084920", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 57, ts 1745879341399735, value "00000002:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 58, ts 1745879341399750, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 59, ts 1745879341399756, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 90, ts 1745879341399761, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 250, ts 1745879341400008, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 252, ts 1745879341400031, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 251, ts 1745879341400056, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 1, fieldId 1, ts 1745879341400081, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 500, ts 1745879341400089, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 66, ts 1745879341400094, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 67, ts 1745879341400339, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 74, ts 1745879341400344, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc0001dc238, pciBandwidthFields5448313761827189385, 0xc0001dc258) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 7, name pciBandwidthFields5448313761827189385, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth5340877252659961707 0xc0001dc268) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 5 name pciBandwidth5340877252659961707 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 5 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 5, eg 1, eid 1 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 5 added eg 1, eid 1. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 5, 7, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000001 (eg 1, entityId 1, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000001 (eg 1, entityId 1, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 1, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 1, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524402220, eg 1, eid 1, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 236, ts 1745879341400657, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524402290, eg 1, eid 1, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 1, fieldId 235, ts 1745879341400722, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 1 0xc0001dc238 2 0xc0007ea000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 7) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 5) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 2 0xc00035a000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 158, ts 1745879341402800, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 159, ts 1745879341406027, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 164, ts 1745879341406212, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 160, ts 1745879341410181, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 163, ts 1745879341410459, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 162, ts 1745879341410465, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 161, ts 1745879341410470, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 2, fieldId 130, ts 1745879341410476, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 54, ts 1745879341411826, value "GPU-6b6a8033-4d7d-264b-9aff-a49e5fba6561", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 85, ts 1745879341411833, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 82, ts 1745879341411854, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 51, ts 1745879341411865, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 50, ts 1745879341411872, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 53, ts 1745879341411888, value "1321720102528", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 57, ts 1745879341411899, value "00000003:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 58, ts 1745879341411906, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 59, ts 1745879341411912, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 90, ts 1745879341411917, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 250, ts 1745879341412164, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 252, ts 1745879341412198, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 251, ts 1745879341412223, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 1, ts 1745879341412247, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 500, ts 1745879341412253, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 66, ts 1745879341412257, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 67, ts 1745879341412400, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 74, ts 1745879341412405, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc0007de900 0xc0001dc2b0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc0001dc2b8, cpuAffFields8822720529353893854, 0xc0001dc2d8) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 8, name cpuAffFields8822720529353893854, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff4023239466423957120 0xc0001dc2e8) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 6 name cpuAff4023239466423957120 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 6 2) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 6, eg 1, eid 2 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 6 added eg 1, eid 2. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 6, 8, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000002 (eg 1, entityId 2, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000002 (eg 1, entityId 2, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000002 (eg 1, entityId 2, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000002 (eg 1, entityId 2, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524003090, eg 1, eid 2, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 72, ts 1745879341412633, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524002f70, eg 1, eid 2, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 70, ts 1745879341412651, value1 16777215, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524002fe0, eg 1, eid 2, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 71, ts 1745879341412667, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524003450, eg 1, eid 2, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 73, ts 1745879341412686, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 2 0xc0001dc2b8 4 0xc0007b3000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 6) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 8) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 2 0xc00026c380) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341413245, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341413406, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341424531, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 2 0xc00035ca00) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 158, ts 1745879341424661, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 159, ts 1745879341424669, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 164, ts 1745879341424905, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 160, ts 1745879341424912, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 163, ts 1745879341425106, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 162, ts 1745879341425111, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 2, fieldId 161, ts 1745879341425116, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 2, fieldId 130, ts 1745879341425121, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 54, ts 1745879341425131, value "GPU-6b6a8033-4d7d-264b-9aff-a49e5fba6561", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 85, ts 1745879341425137, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 82, ts 1745879341425142, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 51, ts 1745879341425147, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 50, ts 1745879341425152, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 53, ts 1745879341425157, value "1321720102528", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 57, ts 1745879341425162, value "00000003:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 58, ts 1745879341425168, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 59, ts 1745879341425174, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 90, ts 1745879341425179, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 250, ts 1745879341425404, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 252, ts 1745879341425424, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 251, ts 1745879341425443, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 2, fieldId 1, ts 1745879341425468, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 500, ts 1745879341425473, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 66, ts 1745879341425478, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 67, ts 1745879341425613, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 74, ts 1745879341425618, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc0001dc300, pciBandwidthFields14885022467662875403, 0xc0001dc310) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 9, name pciBandwidthFields14885022467662875403, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth10884974692609599466 0xc0001dc320) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 7 name pciBandwidth10884974692609599466 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 7 2) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 7, eg 1, eid 2 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 7 added eg 1, eid 2. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 7, 9, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000002 (eg 1, entityId 2, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000002 (eg 1, entityId 2, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 2, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 2, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524003590, eg 1, eid 2, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 236, ts 1745879341425787, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524003600, eg 1, eid 2, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 2, fieldId 235, ts 1745879341425847, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 2 0xc0001dc300 2 0xc0007eea00) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 9) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x9 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 9 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x9 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 7) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 3 0xc000882000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 158, ts 1745879341426814, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 159, ts 1745879341430005, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 164, ts 1745879341430186, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 160, ts 1745879341433971, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 163, ts 1745879341434247, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 162, ts 1745879341434252, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 161, ts 1745879341434257, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 3, fieldId 130, ts 1745879341434263, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 54, ts 1745879341435599, value "GPU-48940724-cca1-d348-c688-8e8cba9c7806", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 85, ts 1745879341435605, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 82, ts 1745879341435622, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 51, ts 1745879341435633, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 50, ts 1745879341435643, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 53, ts 1745879341435657, value "1321920080219", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 57, ts 1745879341435668, value "00000004:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 58, ts 1745879341435674, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 59, ts 1745879341435680, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 90, ts 1745879341435686, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 250, ts 1745879341435929, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 252, ts 1745879341435951, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 251, ts 1745879341435972, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 1, ts 1745879341435996, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 500, ts 1745879341436002, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 66, ts 1745879341436006, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 67, ts 1745879341436132, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 74, ts 1745879341436137, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc0007df600 0xc0001dc36c) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc0001dc380, cpuAffFields7619478093955637007, 0xc0001dc390) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 10, name cpuAffFields7619478093955637007, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff3049006806958683915 0xc0001dc3a0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 8 name cpuAff3049006806958683915 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 8 3) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 8, eg 1, eid 3 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 8 added eg 1, eid 3. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 8, 10, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000003 (eg 1, entityId 3, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000003 (eg 1, entityId 3, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000003 (eg 1, entityId 3, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000003 (eg 1, entityId 3, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524004e90, eg 1, eid 3, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 72, ts 1745879341436383, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5240036b0, eg 1, eid 3, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 70, ts 1745879341436415, value1 16777215, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5240050f0, eg 1, eid 3, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 73, ts 1745879341436444, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524004e00, eg 1, eid 3, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 71, ts 1745879341436468, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 3 0xc0001dc380 4 0xc000886000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 8) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 8 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 10) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xa [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 10 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xa [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 3 0xc00026c540) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341436970, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341437117, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341447678, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 3 0xc000832000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 158, ts 1745879341447817, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 159, ts 1745879341447832, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 164, ts 1745879341448075, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 160, ts 1745879341448082, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 163, ts 1745879341448281, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 162, ts 1745879341448287, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 3, fieldId 161, ts 1745879341448292, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 3, fieldId 130, ts 1745879341448298, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 54, ts 1745879341448307, value "GPU-48940724-cca1-d348-c688-8e8cba9c7806", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 85, ts 1745879341448313, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 82, ts 1745879341448318, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 51, ts 1745879341448323, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 50, ts 1745879341448328, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 53, ts 1745879341448334, value "1321920080219", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 57, ts 1745879341448340, value "00000004:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 58, ts 1745879341448346, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 59, ts 1745879341448351, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 90, ts 1745879341448357, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 250, ts 1745879341448581, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 252, ts 1745879341448602, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 251, ts 1745879341448626, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 3, fieldId 1, ts 1745879341448650, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 500, ts 1745879341448656, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 66, ts 1745879341448660, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 67, ts 1745879341448785, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 74, ts 1745879341448789, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc00052e018, pciBandwidthFields4948538885147978654, 0xc00052e048) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 11, name pciBandwidthFields4948538885147978654, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth9197135987259353481 0xc00052e058) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 9 name pciBandwidth9197135987259353481 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 9 3) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 9, eg 1, eid 3 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 9 added eg 1, eid 3. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 9, 11, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000003 (eg 1, entityId 3, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000003 (eg 1, entityId 3, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 3, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 3, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5240052a0, eg 1, eid 3, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 235, ts 1745879341448976, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524005230, eg 1, eid 3, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 3, fieldId 236, ts 1745879341449383, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 3 0xc00052e018 2 0xc00083a000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 11) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xb [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 11 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xb [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 9) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 9 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 4 0xc000883500) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 158, ts 1745879341450088, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 159, ts 1745879341453289, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 164, ts 1745879341453469, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 160, ts 1745879341457221, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 163, ts 1745879341457495, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 162, ts 1745879341457501, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 161, ts 1745879341457506, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 4, fieldId 130, ts 1745879341457511, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 54, ts 1745879341458798, value "GPU-99d81d30-b0a3-7741-f074-c8db5414e1d5", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 85, ts 1745879341458804, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 82, ts 1745879341458821, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 51, ts 1745879341458831, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 50, ts 1745879341458836, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 53, ts 1745879341458846, value "1322521009087", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 57, ts 1745879341458856, value "0000000B:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 58, ts 1745879341458862, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 59, ts 1745879341458868, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 90, ts 1745879341458874, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 250, ts 1745879341459112, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 252, ts 1745879341459132, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 251, ts 1745879341459154, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 1, ts 1745879341459177, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 500, ts 1745879341459182, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 66, ts 1745879341459187, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 67, ts 1745879341459305, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 74, ts 1745879341459310, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc00021c900 0xc00052e000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc00052e008, cpuAffFields17029925190749189821, 0xc00052e048) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 12, name cpuAffFields17029925190749189821, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff7864888677015435961 0xc00052e068) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 10 name cpuAff7864888677015435961 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 10 4) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 10, eg 1, eid 4 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 10 added eg 1, eid 4. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 10, 12, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000004 (eg 1, entityId 4, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000004 (eg 1, entityId 4, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000004 (eg 1, entityId 4, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000004 (eg 1, entityId 4, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524005350, eg 1, eid 4, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 70, ts 1745879341459545, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524005c00, eg 1, eid 4, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 72, ts 1745879341459572, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524005b50, eg 1, eid 4, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 71, ts 1745879341459600, value1 4294967040, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006450, eg 1, eid 4, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 73, ts 1745879341459615, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 4 0xc00052e008 4 0xc000848000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 10) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 10 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 12) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xc [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 12 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xc [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 4 0xc00026c1c0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341460044, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341460208, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341470912, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 4 0xc000884a00) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 158, ts 1745879341471046, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 159, ts 1745879341471061, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 164, ts 1745879341471292, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 160, ts 1745879341471299, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 163, ts 1745879341471492, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 162, ts 1745879341471497, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 4, fieldId 161, ts 1745879341471502, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 4, fieldId 130, ts 1745879341471508, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 54, ts 1745879341471518, value "GPU-99d81d30-b0a3-7741-f074-c8db5414e1d5", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 85, ts 1745879341471523, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 82, ts 1745879341471528, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 51, ts 1745879341471534, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 50, ts 1745879341471539, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 53, ts 1745879341471544, value "1322521009087", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 57, ts 1745879341471549, value "0000000B:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 58, ts 1745879341471555, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 59, ts 1745879341471561, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 90, ts 1745879341471566, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 250, ts 1745879341471787, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 252, ts 1745879341471806, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 251, ts 1745879341471829, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 4, fieldId 1, ts 1745879341471851, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 500, ts 1745879341471857, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 66, ts 1745879341471862, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 67, ts 1745879341471987, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 74, ts 1745879341471992, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc00052e080, pciBandwidthFields3209782787722091100, 0xc00052e090) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 13, name pciBandwidthFields3209782787722091100, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth3721099999593734986 0xc00052e0a0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 11 name pciBandwidth3721099999593734986 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 11 4) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 11, eg 1, eid 4 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 11 added eg 1, eid 4. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 11, 13, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000004 (eg 1, entityId 4, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000004 (eg 1, entityId 4, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 4, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 4, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006590, eg 1, eid 4, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 236, ts 1745879341472184, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006600, eg 1, eid 4, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 4, fieldId 235, ts 1745879341472247, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 4 0xc00052e080 2 0xc00083c500) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 13) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xd [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 13 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xd [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 11) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 11 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 5 0xc000342000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 158, ts 1745879341473152, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 159, ts 1745879341476340, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 164, ts 1745879341476524, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 160, ts 1745879341480301, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 163, ts 1745879341480576, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 162, ts 1745879341480581, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 161, ts 1745879341480587, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 5, fieldId 130, ts 1745879341480592, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 54, ts 1745879341481937, value "GPU-ca04069e-e7f1-1cc6-0d08-ea6101bba94b", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 85, ts 1745879341481943, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 82, ts 1745879341481959, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 51, ts 1745879341481969, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 50, ts 1745879341481974, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 53, ts 1745879341481985, value "1321920080071", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 57, ts 1745879341481994, value "0000000C:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 58, ts 1745879341482000, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 59, ts 1745879341482006, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 90, ts 1745879341482012, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 250, ts 1745879341482253, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 252, ts 1745879341482273, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 251, ts 1745879341482295, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 1, ts 1745879341482318, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 500, ts 1745879341482323, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 66, ts 1745879341482328, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 67, ts 1745879341482459, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 74, ts 1745879341482464, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc00079c780 0xc0001dc000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc0001dc008, cpuAffFields16141906561246415114, 0xc0001dc028) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 14, name cpuAffFields16141906561246415114, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff13541450221553130295 0xc0001dc038) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 12 name cpuAff13541450221553130295 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 12 5) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 12, eg 1, eid 5 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 12 added eg 1, eid 5. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 12, 14, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000005 (eg 1, entityId 5, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000005 (eg 1, entityId 5, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000005 (eg 1, entityId 5, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000005 (eg 1, entityId 5, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006690, eg 1, eid 5, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 70, ts 1745879341482668, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006b00, eg 1, eid 5, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 71, ts 1745879341482685, value1 4294967040, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006d60, eg 1, eid 5, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 72, ts 1745879341482702, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524006fa0, eg 1, eid 5, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 73, ts 1745879341482717, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 5 0xc0001dc008 4 0xc000532000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 12) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 12 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 14) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xe [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 14 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xe [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 5 0xc0001da000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341483149, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341483295, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341494288, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 5 0xc00033d500) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 158, ts 1745879341494416, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 159, ts 1745879341494431, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 164, ts 1745879341494664, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 160, ts 1745879341494671, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 163, ts 1745879341494864, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 162, ts 1745879341494870, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 5, fieldId 161, ts 1745879341494875, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 5, fieldId 130, ts 1745879341494881, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 54, ts 1745879341494890, value "GPU-ca04069e-e7f1-1cc6-0d08-ea6101bba94b", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 85, ts 1745879341494896, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 82, ts 1745879341494902, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 51, ts 1745879341494907, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 50, ts 1745879341494912, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 53, ts 1745879341494918, value "1321920080071", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 57, ts 1745879341494924, value "0000000C:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 58, ts 1745879341494930, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 59, ts 1745879341494935, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 90, ts 1745879341494941, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 250, ts 1745879341495161, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 252, ts 1745879341495181, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 251, ts 1745879341495204, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 5, fieldId 1, ts 1745879341495227, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 500, ts 1745879341495232, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 66, ts 1745879341495237, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 67, ts 1745879341495359, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 74, ts 1745879341495363, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 2, 0xc0001dc0a0, pciBandwidthFields10729222497512533817, 0xc0001dc0b0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 15, name pciBandwidthFields10729222497512533817, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 pciBandwidth3681989832861940083 0xc0001dc0c0) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 13 name pciBandwidth3681989832861940083 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 13 5) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 13, eg 1, eid 5 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 13 added eg 1, eid 5. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 13, 15, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 2 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100eb00000005 (eg 1, entityId 5, fieldId 235) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 235, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 235, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x100ec00000005 (eg 1, entityId 5, fieldId 236) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 5, fid 236, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 5, fieldId 236, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5240070c0, eg 1, eid 5, fieldId 236 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 236, ts 1745879341495531, value1 16, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524007130, eg 1, eid 5, fieldId 235 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 5, fieldId 235, ts 1745879341495587, value1 4, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 5 0xc0001dc0a0 2 0xc0007ea000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 15) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0xf [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 15 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0xf [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 13) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 13 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 6 0xc000236000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 158, ts 1745879341496452, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 159, ts 1745879341499659, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 6, fieldId 164, ts 1745879341499844, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 6, fieldId 160, ts 1745879341503621, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 6, fieldId 163, ts 1745879341503894, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 6, fieldId 162, ts 1745879341503900, value1 400, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity double eg 1, eid 6, fieldId 161, ts 1745879341503905, value1 100, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6401] [DcgmCacheManager::AppendEntityDouble] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 6, fieldId 130, ts 1745879341503911, valueSize 980, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 54, ts 1745879341505206, value "GPU-b4f100ac-966f-d0b6-2106-3b1acf29a999", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 85, ts 1745879341505212, value "92.00.45.00.03", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 82, ts 1745879341505230, value "G506.0200.00.04", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 51, ts 1745879341505240, value "NVIDIA", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 50, ts 1745879341505245, value "NVIDIA A100-SXM4-40GB", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 53, ts 1745879341505255, value "1321620037406", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 57, ts 1745879341505265, value "0000000D:00:00.0", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 58, ts 1745879341505271, value1 548409566, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 59, ts 1745879341505277, value1 323948766, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 90, ts 1745879341505283, value1 65536, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 250, ts 1745879341505526, value1 40960, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 252, ts 1745879341505546, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 251, ts 1745879341505564, value1 40442, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity string eg 1, eid 6, fieldId 1, ts 1745879341505594, value "570.124.06", cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6552] [DcgmCacheManager::AppendEntityString] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 500, ts 1745879341505600, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 66, ts 1745879341505604, value1 1, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 67, ts 1745879341505731, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 74, ts 1745879341505736, value1 0, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetAllSupportedDevices(dcgmHandle_t pDcgmHandle, unsigned int gpuIdList[DCGM_MAX_NUM_DEVICES], int *count) (2147483647 0xc00087d380 0xc00052e0ec) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:89] [dcgmGetAllSupportedDevices] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupCreate(dcgmHandle_t pDcgmHandle, int numFieldIds, unsigned short *fieldIds, const char *fieldGroupName, dcgmFieldGrp_t *dcgmFieldGroupId) (2147483647 4, 0xc00052e100, cpuAffFields15329645438121343152, 0xc00052e110) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added field group id 16, name cpuAffFields15329645438121343152, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:172] [DcgmFieldGroupManager::AddFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:586] [dcgmFieldGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupCreate(dcgmHandle_t pDcgmHandle, dcgmGroupType_t type, const char *groupName, dcgmGpuGrp_t *pDcgmGrpId) (2147483647 1 cpuAff15792416825237447091 0xc00052e120) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Added GroupId 14 name cpuAff15792416825237447091 for connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:273] [DcgmGroupManager::AddNewGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:177] [dcgmGroupCreate] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupAddDevice(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, unsigned int gpuId) (2147483647 14 6) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddEntityToGroup groupId 14, eg 1, eid 6 added to the group [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:683] [DcgmGroupInfo::AddEntityToGroup] dcgm_level=INFO
2025/04/28 22:29:01 INFO groupId 14 added eg 1, eid 6. ret 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:490] [DcgmGroupManager::AddEntityToGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:193] [dcgmGroupAddDevice] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmWatchFields(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId, dcgmFieldGrp_t fieldGroupId, long long updateFreq, double maxKeepAge, int maxKeepSamples) (2147483647 14, 16, 30000000, 0, 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Got 1 entities and 4 fields [/builds/dcgm/dcgm/dcgmlib/src/DcgmHostEngineHandler.cpp:3734] [DcgmHostEngineHandler::WatchFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004600000006 (eg 1, entityId 6, fieldId 70) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 70, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 6, fieldId 70, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004700000006 (eg 1, entityId 6, fieldId 71) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 71, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 6, fieldId 71, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004800000006 (eg 1, entityId 6, fieldId 72) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 72, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 6, fieldId 72, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding WatchInfo on entityKey 0x1004900000006 (eg 1, entityId 6, fieldId 73) [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2256] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Adding new watcher type 0, connectionId 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3041] [DcgmCacheManager::AddOrUpdateWatcher] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO UpdateWatchFromWatchers gid 1, eid 6, fid 73, minMonitorFreqUsec 30000000, minMaxAgeUsec 33000000, hsw 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3088] [DcgmCacheManager::UpdateWatchFromWatchers] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO AddFieldWatch eg 1, eid 6, fieldId 73, mfu 30000000, msa 0, mka 1, sfu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3232] [DcgmCacheManager::AddEntityFieldWatch] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524007b50, eg 1, eid 6, fieldId 73 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 73, ts 1745879341505950, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524007870, eg 1, eid 6, fieldId 71 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 71, ts 1745879341505966, value1 255, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb524007920, eg 1, eid 6, fieldId 72 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 72, ts 1745879341505989, value1 0, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Preparing to update watchInfo 0x7fb5240071e0, eg 1, eid 6, fieldId 70 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5477] [DcgmCacheManager::ActuallyUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Checking status for gpu 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2398] [DcgmCacheManager::GetGpuStatus] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 70, ts 1745879341506004, value1 -281474976710656, value2 0, cached 1, buffered 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:562] [dcgmWatchFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmUpdateAllFields(dcgmHandle_t pDcgmHandle, int waitForUpdate) (2147483647 1) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO DoOneUpdateAllFields returned 1745879370758193 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2505] [DcgmCacheManager::UpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:409] [dcgmUpdateAllFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetLatestValuesForFields(dcgmHandle_t pDcgmHandle, int gpuId, unsigned short fieldIds[], unsigned int count, dcgmFieldValue_v1 values[]) (2147483647 6 0xc00052e100 4 0xc000851000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:337] [dcgmGetLatestValuesForFields] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmGpuGrp_t groupId) (2147483647 14) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/core/DcgmModuleCore.cpp:270] [DcgmModuleCore::ProcessMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[NvSwitch]] Unknown subcommand: 3 [/builds/dcgm/dcgm/modules/nvswitch/DcgmModuleNvSwitch.cpp:453] [DcgmNs::DcgmModuleNvSwitch::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Enqueueing simple task 'Process message in TaskRunner' for the 0x7fb5082aa1e0 TaskRunner [/builds/dcgm/dcgm/common/TaskRunner.hpp:139] [DcgmNs::TaskRunner::Enqueue] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is consuming 1 tasks from the queue [/builds/dcgm/dcgm/common/TaskRunner.hpp:399] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] TaskRunner is going to run the 'Process message in TaskRunner' task [/builds/dcgm/dcgm/common/TaskRunner.hpp:429] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] Unknown subcommand 3 [/builds/dcgm/dcgm/dcgm_private/modules/profiling/DcgmModuleProfiling.cpp:1907] [DcgmNs::Modules::Profiling::DcgmModuleProfiling::ProcessCoreMessage] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO [[Profiling]] The task 'Process message in TaskRunner' is done [/builds/dcgm/dcgm/common/TaskRunner.hpp:444] [DcgmNs::TaskRunner::Run] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed GroupId 14 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:320] [DcgmGroupManager::RemoveGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:186] [dcgmGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmFieldGroupDestroy(dcgmHandle_t pDcgmHandle, dcgmFieldGrp_t dcgmFieldGroupId) (2147483647 16) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO dcgmFieldGroupDestroy fieldGroupId 0x10 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2729] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Removed field group 16 [/builds/dcgm/dcgm/dcgmlib/src/DcgmFieldGroup.cpp:237] [DcgmFieldGroupManager::RemoveFieldGroup] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO tsapiFieldGroupDestroy ret 0, fieldGroupId 0x10 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:2744] [tsapiFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:600] [dcgmFieldGroupDestroy] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceTopology(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceTopology_t *deviceTopology) (2147483647 6 0xc00026c380) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 60 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 60 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 60, ts 1745879341506435, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 61 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 61 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForGpu = 0 migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:13641] [DcgmCacheManager::UpdateNvLinkLinkState] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7 has migIsEnabledForAnyGpu 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:462] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 0. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 1. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 2. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 3. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 4. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 5. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 6. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 7. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 8. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 9. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 10. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, NvLink laneId 11. UP [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:508] [UpdateNvLinkLinkStateFromNvml] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 0 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 1 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 2 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 2 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 3 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 3 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 4 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 4 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 5 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 5 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 6 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 6 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Getting CONNECTED_LINK_COUNT for GPU 7 [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:710] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GPU 7 has 12 active NvSwitch NvLinks. [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:730] [HelperGetActiveNvSwitchNvLinkCountsForAllGpusUsingNVML] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 0, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 1, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 2, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 3, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 4, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 5, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 6, localNvLinkQuantity 12, localLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:631] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO gpuId 7, remoteNvLinkMask xfff [/builds/dcgm/dcgm/dcgmlib/src/DcgmTopology.cpp:674] [PopulateTopologyNvLink] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 61, ts 1745879341506578, valueSize 568, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO helperGetTopologyPci returned 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmApi.cpp:3317] [helperGetTopologyPci] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO GetGroupEntities got 8 entities for dynamic group 0 [/builds/dcgm/dcgm/dcgmlib/src/DcgmGroupManager.cpp:411] [DcgmGroupManager::GetGroupEntities] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixing entityGroupId for global field [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:4618] [DcgmCacheManager::GetLatestSample] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO watch key eg 0, eid 0, fieldId 62 doesn't exist. createIfNotExists == false [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:2248] [DcgmCacheManager::GetEntityWatchInfo] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO PrecheckWatchInfoForSamples: not watched [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:3784] [DcgmCacheManager::PrecheckWatchInfoForSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Fixed entityGroupId to be DCGM_FE_NONE fieldId 62 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:5677] [DcgmCacheManager::GetMultipleLatestLiveSamples] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity blob eg 1, eid 0, fieldId 62, ts 1745879341521075, valueSize 2312, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6607] [DcgmCacheManager::AppendEntityBlob] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Returning 0 [/builds/dcgm/dcgm/dcgmlib/entry_point.h:715] [dcgmGetDeviceTopology] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Entering dcgmGetDeviceAttributes(dcgmHandle_t pDcgmHandle, unsigned int gpuId, dcgmDeviceAttributes_t *pDcgmDeviceAttr) (2147483647 6 0xc00035a000) [/builds/dcgm/dcgm/dcgmlib/entry_point.h:97] [dcgmGetDeviceAttributes] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 158, ts 1745879341521210, value1 89, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=DEBUG
2025/04/28 22:29:01 INFO Appended entity i64 eg 1, eid 6, fieldId 159, ts 1745879341521222, value1 92, value2 0, cached 0, buffered 1 [/builds/dcgm/dcgm/dcgmlib/src/DcgmCacheManager.cpp:6498] [DcgmCacheManager::AppendEntityInt64] dcgm_level=D
Raw
exporter-4.2.0-4.1.0.sh
View raw

(Sorry about that, but we can’t show files that are this big right now.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment