Skip to content

Instantly share code, notes, and snippets.

@thearyanahmed

thearyanahmed/gist:11e811986b603f5c225e7dab8fc66543

Created June 10, 2026 18:34
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save thearyanahmed/11e811986b603f5c225e7dab8fc66543 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save thearyanahmed/11e811986b603f5c225e7dab8fc66543 to your computer and use it in GitHub Desktop.
Download ZIP
device_metric_exporter_e2e_logs.txt
Raw
gistfile1.txt
[I] 18:47 /device-metrics-exporter [fix/cper-stale-fatal-health] make docker-shell
Makefile.package:162: warning: overriding commands for target `debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old commands for target `debpkg-sriov-clean'
docker run --rm -it --privileged \
--name paryan_exporter-bld \
-e "USER_NAME=paryan" \
-e "USER_UID=502" \
-e "USER_GID=20" \
-e "GIT_COMMIT=e294fdbd" \
-e "GIT_VERSION=" \
-e "BUILD_DATE=2026-06-10T18:47:30+0100" \
-v /Users/paryan/dev/digitalocean/device-metrics-exporter:/usr/src/github.com/ROCm/device-metrics-exporter \
-w /usr/src/github.com/ROCm/device-metrics-exporter \
docker.io/rocm/device-metrics-exporter-build:v1.10 \
bash -c "cd /usr/src/github.com/ROCm/device-metrics-exporter && git config --global --add safe.directory /usr/src/github.com/ROCm/device-metrics-exporter && bash"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
+ set -euo pipefail
+ dir=/usr/src/github.com/ROCm/device-metrics-exporter
+ netns=/var/run/netns
+ PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ dockerd -s vfs
+ trap term INT TERM
+ mkdir -p /var/run/netns
+ rm -f /usr/src/github.com/ROCm/device-metrics-exporter/.container_ready
+ export GOFLAGS=-mod=vendor
+ GOFLAGS=-mod=vendor
+ sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
+ [[ -n paryan ]]
+ [[ -n 502 ]]
+ [[ -n 20 ]]
+ echo 'Creating user paryan with UID=502, GID=20...'
Creating user paryan with UID=502, GID=20...
+ getent group 20
+ id -u paryan
+ useradd -m -u 502 -g 20 -s /bin/bash paryan
+ echo 'paryan ALL=(ALL) NOPASSWD:ALL'
+ chmod 0440 /etc/sudoers.d/paryan
+ getent group docker
+ usermod -aG docker paryan
+ chown -R 502:20 /usr/src/github.com/ROCm/device-metrics-exporter
INFO[2026-06-10T17:47:31.841819655Z] Starting up
INFO[2026-06-10T17:47:31.849023100Z] containerd not running, starting managed containerd
INFO[2026-06-10T17:47:31.855842007Z] started new containerd process address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=43
+ chown -R 502:20 /home/paryan
+ su - paryan -c 'echo '\''export GOPATH=/home/paryan/go'\'' >> ~/.bashrc'
INFO[2026-06-10T17:47:32.011930825Z] starting containerd revision=193637f7ee8ae5f5aa5248f49e7baa3e6164966e version=v2.2.4
WARN[2026-06-10T17:47:32.024046400Z] Configuration migrated from version 2, use `containerd config migrate` to avoid migration t="619.746µs"
INFO[2026-06-10T17:47:32.024249899Z] loading plugin id=io.containerd.content.v1.content type=io.containerd.content.v1
INFO[2026-06-10T17:47:32.024915935Z] loading plugin id=io.containerd.image-verifier.v1.bindir type=io.containerd.image-verifier.v1
INFO[2026-06-10T17:47:32.025011810Z] loading plugin id=io.containerd.internal.v1.opt type=io.containerd.internal.v1
INFO[2026-06-10T17:47:32.025889178Z] loading plugin id=io.containerd.warning.v1.deprecations type=io.containerd.warning.v1
INFO[2026-06-10T17:47:32.025937886Z] loading plugin id=io.containerd.mount-handler.v1.erofs type=io.containerd.mount-handler.v1
INFO[2026-06-10T17:47:32.025987636Z] loading plugin id=io.containerd.snapshotter.v1.blockfile type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026346341Z] skip loading plugin error="no scratch file generator: skip plugin" id=io.containerd.snapshotter.v1.blockfile type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026402632Z] loading plugin id=io.containerd.snapshotter.v1.btrfs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026720130Z] skip loading plugin error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" id=io.containerd.snapshotter.v1.btrfs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026756671Z] loading plugin id=io.containerd.snapshotter.v1.devmapper type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026838546Z] skip loading plugin error="devmapper not configured: skip plugin" id=io.containerd.snapshotter.v1.devmapper type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.026850962Z] loading plugin id=io.containerd.snapshotter.v1.erofs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.028110120Z] loading plugin id=io.containerd.snapshotter.v1.native type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.028328160Z] loading plugin id=io.containerd.snapshotter.v1.overlayfs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.029221403Z] loading plugin id=io.containerd.snapshotter.v1.zfs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.029367277Z] skip loading plugin error="lstat /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs: no such file or directory: skip plugin" id=io.containerd.snapshotter.v1.zfs type=io.containerd.snapshotter.v1
INFO[2026-06-10T17:47:32.029376360Z] loading plugin id=io.containerd.event.v1.exchange type=io.containerd.event.v1
INFO[2026-06-10T17:47:32.029490776Z] loading plugin id=io.containerd.monitor.task.v1.cgroups type=io.containerd.monitor.task.v1
INFO[2026-06-10T17:47:32.030345728Z] loading plugin id=io.containerd.metadata.v1.bolt type=io.containerd.metadata.v1
INFO[2026-06-10T17:47:32.030644975Z] metadata content store policy set policy=shared
INFO[2026-06-10T17:47:32.045669612Z] loading plugin id=io.containerd.gc.v1.scheduler type=io.containerd.gc.v1
INFO[2026-06-10T17:47:32.046455731Z] loading plugin id=io.containerd.nri.v1.nri type=io.containerd.nri.v1
INFO[2026-06-10T17:47:32.047026060Z] built-in NRI default validator is disabled
INFO[2026-06-10T17:47:32.047076059Z] runtime interface created
INFO[2026-06-10T17:47:32.047079726Z] created NRI interface
INFO[2026-06-10T17:47:32.047091309Z] loading plugin id=io.containerd.differ.v1.erofs type=io.containerd.differ.v1
INFO[2026-06-10T17:47:32.047751304Z] skip loading plugin error="failed to check mkfs.erofs availability: failed to run mkfs.erofs --help: exec: \"mkfs.erofs\": executable file not found in $PATH: skip plugin" id=io.containerd.differ.v1.erofs type=io.containerd.differ.v1
INFO[2026-06-10T17:47:32.047763887Z] loading plugin id=io.containerd.mount-manager.v1.bolt type=io.containerd.mount-manager.v1
INFO[2026-06-10T17:47:32.051152987Z] loading plugin id=io.containerd.differ.v1.walking type=io.containerd.differ.v1
INFO[2026-06-10T17:47:32.051308777Z] loading plugin id=io.containerd.lease.v1.manager type=io.containerd.lease.v1
INFO[2026-06-10T17:47:32.051481318Z] loading plugin id=io.containerd.service.v1.containers-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051542401Z] loading plugin id=io.containerd.service.v1.content-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051576400Z] loading plugin id=io.containerd.service.v1.diff-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051668358Z] loading plugin id=io.containerd.service.v1.images-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051721816Z] loading plugin id=io.containerd.service.v1.introspection-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051811440Z] loading plugin id=io.containerd.service.v1.namespaces-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051846065Z] loading plugin id=io.containerd.service.v1.snapshots-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.051881648Z] loading plugin id=io.containerd.shim.v1.manager type=io.containerd.shim.v1
INFO[2026-06-10T17:47:32.051979356Z] loading plugin id=io.containerd.runtime.v2.task type=io.containerd.runtime.v2
INFO[2026-06-10T17:47:32.052833849Z] loading plugin id=io.containerd.service.v1.tasks-service type=io.containerd.service.v1
INFO[2026-06-10T17:47:32.053194305Z] loading plugin id=io.containerd.grpc.v1.containers type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053236054Z] loading plugin id=io.containerd.grpc.v1.content type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053270137Z] loading plugin id=io.containerd.grpc.v1.diff type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053299221Z] loading plugin id=io.containerd.grpc.v1.events type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053321304Z] loading plugin id=io.containerd.grpc.v1.images type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053363387Z] loading plugin id=io.containerd.grpc.v1.introspection type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053409970Z] loading plugin id=io.containerd.grpc.v1.leases type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053448053Z] loading plugin id=io.containerd.grpc.v1.mounts type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053473011Z] loading plugin id=io.containerd.grpc.v1.namespaces type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.053517219Z] loading plugin id=io.containerd.sandbox.store.v1.local type=io.containerd.sandbox.store.v1
INFO[2026-06-10T17:47:32.053542427Z] loading plugin id=io.containerd.transfer.v1.local type=io.containerd.transfer.v1
INFO[2026-06-10T17:47:32.054216297Z] loading plugin id=io.containerd.cri.v1.images type=io.containerd.cri.v1
INFO[2026-06-10T17:47:32.055509870Z] Get image filesystem path "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs" for snapshotter "overlayfs"
INFO[2026-06-10T17:47:32.055594578Z] Start snapshots syncer
INFO[2026-06-10T17:47:32.055684078Z] loading plugin id=io.containerd.cri.v1.runtime type=io.containerd.cri.v1
INFO[2026-06-10T17:47:32.057006693Z] starting cri plugin config="{\"containerd\":{\"defaultRuntimeName\":\"runc\",\"runtimes\":{\"runc\":{\"runtimeType\":\"io.containerd.runc.v2\",\"runtimePath\":\"\",\"PodAnnotations\":null,\"ContainerAnnotations\":null,\"options\":{\"BinaryName\":\"\",\"CriuImagePath\":\"\",\"CriuWorkPath\":\"\",\"IoGid\":0,\"IoUid\":0,\"NoNewKeyring\":false,\"Root\":\"\",\"ShimCgroup\":\"\",\"SystemdCgroup\":false},\"privileged_without_host_devices\":false,\"privileged_without_host_devices_all_devices_allowed\":false,\"cgroupWritable\":false,\"baseRuntimeSpec\":\"\",\"cniConfDir\":\"\",\"cniMaxConfNum\":0,\"snapshotter\":\"\",\"sandboxer\":\"podsandbox\",\"io_type\":\"\"}},\"ignoreBlockIONotEnabledErrors\":false,\"ignoreRdtNotEnabledErrors\":false},\"cni\":{\"binDir\":\"\",\"binDirs\":[\"/opt/cni/bin\"],\"confDir\":\"/etc/cni/net.d\",\"maxConfNum\":1,\"setupSerially\":false,\"confTemplate\":\"\",\"ipPref\":\"\",\"useInternalLoopback\":false},\"enableSelinux\":false,\"selinuxCategoryRange\":1024,\"maxContainerLogLineSize\":16384,\"disableApparmor\":false,\"restrictOOMScoreAdj\":false,\"disableProcMount\":false,\"unsetSeccompProfile\":\"\",\"tolerateMissingHugetlbController\":true,\"disableHugetlbController\":true,\"device_ownership_from_security_context\":false,\"ignoreImageDefinedVolumes\":false,\"netnsMountsUnderStateDir\":false,\"enableUnprivilegedPorts\":true,\"enableUnprivilegedICMP\":true,\"enableCDI\":true,\"cdiSpecDirs\":[\"/etc/cdi\",\"/var/run/cdi\"],\"drainExecSyncIOTimeout\":\"0s\",\"ignoreDeprecationWarnings\":null,\"containerdRootDir\":\"/var/lib/docker/containerd/daemon\",\"containerdEndpoint\":\"/var/run/docker/containerd/containerd.sock\",\"rootDir\":\"/var/lib/docker/containerd/daemon/io.containerd.grpc.v1.cri\",\"stateDir\":\"/var/run/docker/containerd/daemon/io.containerd.grpc.v1.cri\"}"
INFO[2026-06-10T17:47:32.057604521Z] loading plugin id=io.containerd.podsandbox.controller.v1.podsandbox type=io.containerd.podsandbox.controller.v1
INFO[2026-06-10T17:47:32.058103934Z] loading plugin id=io.containerd.sandbox.controller.v1.shim type=io.containerd.sandbox.controller.v1
INFO[2026-06-10T17:47:32.058253225Z] loading plugin id=io.containerd.grpc.v1.sandbox-controllers type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058379890Z] loading plugin id=io.containerd.grpc.v1.sandboxes type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058417807Z] loading plugin id=io.containerd.grpc.v1.snapshots type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058438390Z] loading plugin id=io.containerd.streaming.v1.manager type=io.containerd.streaming.v1
INFO[2026-06-10T17:47:32.058466556Z] loading plugin id=io.containerd.grpc.v1.streaming type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058495140Z] loading plugin id=io.containerd.grpc.v1.tasks type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058528264Z] loading plugin id=io.containerd.grpc.v1.transfer type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058604430Z] loading plugin id=io.containerd.grpc.v1.version type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.058625472Z] loading plugin id=io.containerd.monitor.container.v1.restart type=io.containerd.monitor.container.v1
INFO[2026-06-10T17:47:32.058948886Z] loading plugin id=io.containerd.tracing.processor.v1.otlp type=io.containerd.tracing.processor.v1
INFO[2026-06-10T17:47:32.059163593Z] skip loading plugin error="skip plugin: tracing endpoint not configured" id=io.containerd.tracing.processor.v1.otlp type=io.containerd.tracing.processor.v1
INFO[2026-06-10T17:47:32.059182259Z] loading plugin id=io.containerd.internal.v1.tracing type=io.containerd.internal.v1
INFO[2026-06-10T17:47:32.059256634Z] skip loading plugin error="skip plugin: tracing endpoint not configured" id=io.containerd.internal.v1.tracing type=io.containerd.internal.v1
INFO[2026-06-10T17:47:32.059275092Z] loading plugin id=io.containerd.ttrpc.v1.otelttrpc type=io.containerd.ttrpc.v1
INFO[2026-06-10T17:47:32.059295759Z] loading plugin id=io.containerd.grpc.v1.healthcheck type=io.containerd.grpc.v1
INFO[2026-06-10T17:47:32.061637824Z] serving... address=/var/run/docker/containerd/containerd-debug.sock
INFO[2026-06-10T17:47:32.061702240Z] serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2026-06-10T17:47:32.061733948Z] serving... address=/var/run/docker/containerd/containerd.sock
INFO[2026-06-10T17:47:32.062809274Z] containerd successfully booted in 0.055518s
INFO[2026-06-10T17:47:32.087459587Z] OTEL tracing is not configured, using no-op tracer provider
INFO[2026-06-10T17:47:32.088938076Z] CDI directory does not exist, skipping dir=/var/run/cdi
INFO[2026-06-10T17:47:32.089048450Z] CDI directory does not exist, skipping dir=/etc/cdi
+ su - paryan -c 'echo '\''export PATH=$GOPATH/bin:/usr/local/go/bin:$PATH'\'' >> ~/.bashrc'
INFO[2026-06-10T17:47:32.109250797Z] Creating a containerd client address=/var/run/docker/containerd/containerd.sock timeout=1m0s
INFO[2026-06-10T17:47:32.131564837Z] Loading containers: start.
INFO[2026-06-10T17:47:32.131800418Z] NRI is disabled
INFO[2026-06-10T17:47:32.131932750Z] [graphdriver] trying configured driver: vfs
INFO[2026-06-10T17:47:32.135218434Z] Restoring containers: start.
INFO[2026-06-10T17:47:32.161854941Z] Deleting nftables IPv4 rules error="exit status 1" output="Error: Could not process rule: No such file or directory\ndelete table ip docker-bridges"
+ su - paryan -c 'echo '\''export PATH=/usr/local/go/bin:$PATH'\'' >> ~/.bashrc'
INFO[2026-06-10T17:47:32.179932304Z] Deleting nftables IPv6 rules error="exit status 1" output="Error: Could not process rule: No such file or directory\ndelete table ip6 docker-bridges"
+ exec su - paryan -c bash -c 'cd /usr/src/github.com/ROCm/device-metrics-exporter && git config --global --add safe.directory /usr/src/github.com/ROCm/device-metrics-exporter && bash'
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$ INFO[2026-06-10T17:47:34.192817444Z] Loading containers: done.
INFO[2026-06-10T17:47:34.209163779Z] Docker daemon commit=285b471 containerd-snapshotter=false storage-driver=vfs version=29.5.3
INFO[2026-06-10T17:47:34.209837107Z] Initializing buildkit
INFO[2026-06-10T17:47:34.257838452Z] Completed buildkit initialization
INFO[2026-06-10T17:47:34.265271354Z] Daemon has completed initialization
INFO[2026-06-10T17:47:34.265456936Z] API listen on /var/run/docker.sock
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$ sudo chown -R 502:20 .
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$ make e2e
Makefile.package:162: warning: overriding recipe for target 'debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old recipe for target 'debpkg-sriov-clean'
make docker-mock
make[1]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter'
Makefile.package:162: warning: overriding recipe for target 'debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old recipe for target 'debpkg-sriov-clean'
Generating config-nic.json from config.json
Generating config-gpu.json from config.json
Config generation complete
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go: downloading google.golang.org/protobuf v1.34.2
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1
go: downloading google.golang.org/grpc v1.5.1
go: downloading google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.5.1
go: downloading google.golang.org/protobuf v1.34.1
go install go.uber.org/mock/mockgen@v0.6.0
go: downloading go.uber.org/mock v0.6.0
go: downloading golang.org/x/mod v0.27.0
go: downloading golang.org/x/tools v0.36.0
go: downloading golang.org/x/sync v0.16.0
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.64.8
go: downloading github.com/golangci/golangci-lint v1.64.8
go: downloading github.com/gofrs/flock v0.12.1
go: downloading github.com/fatih/color v1.18.0
go: downloading github.com/hashicorp/go-version v1.7.0
go: downloading github.com/pelletier/go-toml/v2 v2.2.3
go: downloading github.com/santhosh-tekuri/jsonschema/v6 v6.0.1
go: downloading github.com/spf13/cobra v1.9.1
go: downloading github.com/spf13/pflag v1.0.6
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/spf13/viper v1.12.0
go: downloading go.uber.org/automaxprocs v1.6.0
go: downloading golang.org/x/tools v0.31.0
go: downloading github.com/go-viper/mapstructure/v2 v2.2.1
go: downloading github.com/ldez/grignotin v0.9.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/pelletier/go-toml v1.9.5
go: downloading golang.org/x/mod v0.24.0
go: downloading github.com/golangci/plugin-module-register v0.1.1
go: downloading github.com/mattn/go-colorable v0.1.14
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/stretchr/testify v1.10.0
go: downloading github.com/go-xmlfmt/xmlfmt v1.1.3
go: downloading github.com/golangci/revgrep v0.8.0
go: downloading github.com/rogpeppe/go-internal v1.14.1
go: downloading golang.org/x/sys v0.31.0
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading golang.org/x/text v0.22.0
go: downloading github.com/alingse/asasalint v0.0.11
go: downloading github.com/tdakkota/asciicheck v0.4.1
go: downloading github.com/breml/bidichk v0.3.2
go: downloading github.com/timakin/bodyclose v0.0.0-20241017074812-ed6a65f985e3
go: downloading github.com/sivchari/containedctx v1.0.3
go: downloading github.com/lasiar/canonicalheader v1.1.2
go: downloading github.com/kkHAIKE/contextcheck v1.1.6
go: downloading github.com/karamaru-alpha/copyloopvar v1.2.1
go: downloading github.com/bkielbasa/cyclop v1.2.3
go: downloading gitlab.com/bosi/decorder v0.4.2
go: downloading github.com/OpenPeeDeeP/depguard/v2 v2.2.1
go: downloading github.com/Abirdcfly/dupword v0.1.3
go: downloading github.com/golangci/dupl v0.0.0-20250308024227-f665c8d69b32
go: downloading github.com/Djarvur/go-err113 v0.0.0-20210108212216-aea10b59be24
go: downloading github.com/kisielk/errcheck v1.9.0
go: downloading github.com/breml/errchkjson v0.4.0
go: downloading github.com/Antonboom/errname v1.0.0
go: downloading github.com/charithe/durationcheck v0.0.10
go: downloading github.com/polyfloyd/go-errorlint v1.7.1
go: downloading github.com/GaijinEntertainment/go-exhaustruct/v3 v3.3.1
go: downloading github.com/ldez/exptostd v0.4.2
go: downloading github.com/nishanths/exhaustive v0.12.0
go: downloading github.com/Crocmagnon/fatcontext v0.7.1
go: downloading github.com/gostaticanalysis/forcetypeassert v0.2.0
go: downloading github.com/ashanbrown/forbidigo v1.6.0
go: downloading github.com/ultraware/funlen v0.2.0
go: downloading github.com/nunnatsa/ginkgolinter v0.19.1
go: downloading 4d63.com/gocheckcompilerdirectives v1.3.0
go: downloading 4d63.com/gochecknoglobals v0.2.2
go: downloading github.com/alecthomas/go-check-sumtype v0.3.1
go: downloading github.com/jgautheron/goconst v1.7.1
go: downloading github.com/uudashr/gocognit v1.2.0
go: downloading github.com/fzipp/gocyclo v0.6.0
go: downloading github.com/go-critic/go-critic v0.12.0
go: downloading github.com/quasilyte/go-ruleguard/dsl v0.3.22
go: downloading github.com/tetafro/godot v1.5.0
go: downloading github.com/matoous/godox v1.1.0
go: downloading github.com/denis-tingaikin/go-header v0.5.0
go: downloading github.com/ldez/gomoddirectives v0.6.1
go: downloading github.com/ryancurrah/gomodguard v1.3.5
go: downloading github.com/golangci/go-printf-func-name v0.1.0
go: downloading honnef.co/go/tools v0.6.1
go: downloading github.com/securego/gosec/v2 v2.22.2
go: downloading github.com/quasilyte/go-ruleguard v0.4.3-0.20240823090925-0fe6f58b47b1
go: downloading github.com/xen0n/gosmopolitan v1.2.2
go: downloading github.com/leonklingele/grouper v1.1.2
go: downloading github.com/uudashr/iface v1.3.1
go: downloading github.com/julz/importas v0.2.0
go: downloading github.com/macabu/inamedparam v0.1.3
go: downloading github.com/gordonklaus/ineffassign v0.1.0
go: downloading github.com/sashamelentyev/interfacebloat v1.1.0
go: downloading github.com/ckaznocha/intrange v0.3.0
go: downloading github.com/butuzov/ireturn v0.3.1
go: downloading github.com/timonwong/loggercheck v0.10.1
go: downloading github.com/yagipy/maintidx v1.0.0
go: downloading github.com/ashanbrown/makezero v1.2.0
go: downloading github.com/butuzov/mirror v1.3.0
go: downloading github.com/golangci/misspell v0.6.0
go: downloading github.com/tommy-muehle/go-mnd/v2 v2.5.1
go: downloading github.com/alexkohler/nakedret/v2 v2.0.5
go: downloading go-simpler.org/musttag v0.13.0
go: downloading github.com/nakabonne/nestif v0.3.1
go: downloading github.com/gostaticanalysis/nilerr v0.1.1
go: downloading github.com/alingse/nilnesserr v0.1.2
go: downloading github.com/Antonboom/nilnil v1.0.1
go: downloading github.com/ssgreg/nlreturn/v2 v2.2.1
go: downloading github.com/sonatard/noctx v0.1.0
go: downloading github.com/firefart/nonamedreturns v1.0.5
go: downloading github.com/stbenjam/no-sprintf-host-port v0.2.0
go: downloading github.com/kunwardeep/paralleltest v1.0.10
go: downloading github.com/catenacyber/perfsprint v0.8.2
go: downloading github.com/alexkohler/prealloc v1.0.0
go: downloading github.com/nishanths/predeclared v0.2.2
go: downloading github.com/yeya24/promlinter v0.3.0
go: downloading github.com/ghostiam/protogetter v0.3.9
go: downloading github.com/curioswitch/go-reassign v0.3.0
go: downloading github.com/raeperd/recvcheck v0.2.0
go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c
go: downloading github.com/mgechev/revive v1.7.0
go: downloading github.com/jingyugao/rowserrcheck v1.1.1
go: downloading go-simpler.org/sloglint v0.9.0
go: downloading github.com/jjti/go-spancheck v0.6.4
go: downloading github.com/ryanrolds/sqlclosecheck v0.5.1
go: downloading github.com/4meepo/tagalign v1.4.2
go: downloading github.com/ldez/tagliatelle v0.7.1
go: downloading github.com/sivchari/tenv v1.12.1
go: downloading github.com/maratori/testableexamples v1.0.0
go: downloading github.com/Antonboom/testifylint v1.5.2
go: downloading github.com/kulti/thelper v0.6.3
go: downloading github.com/maratori/testpackage v1.1.1
go: downloading github.com/moricho/tparallel v0.3.2
go: downloading github.com/golangci/unconvert v0.0.0-20240309020433-c5143eacb3ed
go: downloading mvdan.cc/unparam v0.0.0-20240528143540-8a5130ca722f
go: downloading github.com/sashamelentyev/usestdlibvars v1.28.0
go: downloading github.com/ldez/usetesting v0.4.2
go: downloading github.com/blizzy78/varnamelen v0.8.0
go: downloading github.com/sanposhiho/wastedassign/v2 v2.1.0
go: downloading github.com/ultraware/whitespace v0.2.0
go: downloading github.com/tomarrell/wrapcheck/v2 v2.10.0
go: downloading github.com/bombsimon/wsl/v4 v4.5.0
go: downloading github.com/ykadowak/zerologlint v0.1.5
go: downloading github.com/fsnotify/fsnotify v1.5.4
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/spf13/afero v1.12.0
go: downloading github.com/spf13/cast v1.5.0
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading github.com/daixiang0/gci v0.13.5
go: downloading github.com/golangci/gofmt v0.0.0-20250106114630-d62b90e6713d
go: downloading mvdan.cc/gofumpt v0.7.0
go: downloading github.com/sourcegraph/go-diff v0.7.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/stretchr/objx v0.5.2
go: downloading golang.org/x/sync v0.12.0
go: downloading github.com/go-toolsmith/astcast v1.1.0
go: downloading github.com/gostaticanalysis/analysisutil v0.7.1
go: downloading github.com/gobwas/glob v0.2.3
go: downloading github.com/google/go-cmp v0.7.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/go-toolsmith/astcopy v1.1.0
go: downloading github.com/go-toolsmith/astequal v1.2.0
go: downloading github.com/go-toolsmith/astfmt v1.1.0
go: downloading github.com/go-toolsmith/astp v1.1.0
go: downloading github.com/go-toolsmith/strparse v1.1.0
go: downloading github.com/go-toolsmith/typep v1.1.0
go: downloading github.com/quasilyte/regex/syntax v0.0.0-20210819130434-b3f0c404a727
go: downloading github.com/Masterminds/semver/v3 v3.3.0
go: downloading github.com/ccojocar/zxcvbn-go v1.0.2
go: downloading github.com/gostaticanalysis/comment v1.5.0
go: downloading github.com/prometheus/client_golang v1.12.1
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/fatih/structtag v1.2.0
go: downloading github.com/ettle/strcase v0.2.0
go: downloading github.com/hashicorp/go-immutable-radix/v2 v2.1.0
go: downloading github.com/subosito/gotenv v1.4.1
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.6
go: downloading github.com/hexops/gotextdiff v1.0.3
go: downloading go.uber.org/zap v1.24.0
go: downloading github.com/quasilyte/gogrep v0.5.0
go: downloading github.com/quasilyte/stdinfo v0.0.0-20220114132959-f7386bf02567
go: downloading golang.org/x/exp/typeparams v0.0.0-20250210185358-939b2ce775ac
go: downloading github.com/chavacava/garif v0.1.0
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/golang/protobuf v1.5.3
go: downloading github.com/hashicorp/golang-lru/v2 v2.0.7
go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading github.com/prometheus/common v0.32.1
go: downloading github.com/prometheus/procfs v0.7.3
go: downloading google.golang.org/protobuf v1.36.5
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading github.com/mattn/go-runewidth v0.0.16
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/rivo/uniseg v0.4.7
go install golang.org/x/tools/cmd/goimports@v0.40.0
go: downloading golang.org/x/tools v0.40.0
go: downloading golang.org/x/telemetry v0.0.0-20251203150158-8fff8a5912fc
go: downloading golang.org/x/mod v0.31.0
go: downloading golang.org/x/sync v0.19.0
go install github.com/alta/protopatch/cmd/protoc-gen-go-patch@v0.5.3
go: downloading github.com/alta/protopatch v0.5.3
go: downloading google.golang.org/protobuf v1.31.0
go: downloading golang.org/x/tools v0.12.0
+++++++++++++++++ Generating pkg/testrunner/proto +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/testrunner/proto'
building proto
+++++++++++++++++ Generating config.proto
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/testrunner/proto'
+++++++++++++++++ Generating pkg/amdgpu/proto +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/proto'
building proto
+++++++++++++++++ Generating events.proto
+++++++++++++++++ Generating gpu.proto
+++++++++++++++++ Generating gpumetricssvc.proto
+++++++++++++++++ Generating luaplugin.proto
+++++++++++++++++ Generating rocprofiler.proto
+++++++++++++++++ Generating testsvc.proto
+++++++++++++++++ Generating types.proto
+++++++++++++++++ Generating ual.proto
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/proto'
+++++++++++++++++ Generating pkg/exporter/proto +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/exporter/proto'
building proto
+++++++++++++++++ Generating exporterconfig.proto
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/exporter/proto'
+++++++++++++++++ Generating pkg/amdnic/proto +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdnic/proto'
building proto
+++++++++++++++++ Generating ethtool.proto +++++++++++++++++
+++++++++++++++++ Generating nic.proto +++++++++++++++++
+++++++++++++++++ Generating nicmetricssvc.proto +++++++++++++++++
+++++++++++++++++ Generating rdmastats.proto +++++++++++++++++
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdnic/proto'
+++++++++++++++++ Generating mock pkg/amdgpu/mock +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock'
building mock gen
+++++++++++++++++ Generating mock gpu +++++++++++++++++
+++++++++++++++++ Generating mock events +++++++++++++++++
+++++++++++++++++ Generating mock ual +++++++++++++++++
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_events.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_gpu.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_scheduler.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_ual.go
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock'
+++++++++++++++++ Generating mock pkg/exporter/scheduler +++++++++++++++++
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/exporter/scheduler'
building mock scheduler
+++++++++++++++++ Generating mock scheduler +++++++++++++++++
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_events.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_gpu.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_scheduler.go
/usr/src/github.com/ROCm/device-metrics-exporter/pkg/amdgpu/mock_gen/mock_ual.go
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/pkg/exporter/scheduler'
GO_BUILD_TAGS=mock make amdexporter
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter'
Makefile.package:162: warning: overriding recipe for target 'debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old recipe for target 'debpkg-sriov-clean'
building metrics client
CGO_ENABLED=0 go build -C tools/metricsclient -o /usr/src/github.com/ROCm/device-metrics-exporter/bin/metricsclient
building amd gpu health util
CGO_ENABLED=0 go build -C tools/amd-gpu-health -o /usr/src/github.com/ROCm/device-metrics-exporter/bin/amdgpuhealth
building amd metrics exporter
CGO_ENABLED=0 go build -C cmd/exporter \
-tags mock \
-ldflags "-X main.Version=fix/cper-stale-fatal-health \
-X main.GitCommit=e294fdbd \
-X main.BuildDate=2026-06-10T17:54:24+0000 \
-X main.Publish=" \
-o /usr/src/github.com/ROCm/device-metrics-exporter/bin/amd-metrics-exporter
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter'
make mock-rocpctl
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter'
Makefile.package:162: warning: overriding recipe for target 'debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old recipe for target 'debpkg-sriov-clean'
make -C tools/mock-rocpctl BIN_PATH=/usr/src/github.com/ROCm/device-metrics-exporter/bin
make[3]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/tools/mock-rocpctl'
g++ -o /usr/src/github.com/ROCm/device-metrics-exporter/bin/rocpctl-mock rocpctl.cpp
make[3]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/tools/mock-rocpctl'
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter'
make -C docker TOP_DIR=/usr/src/github.com/ROCm/device-metrics-exporter EXPORTER_IMAGE_NAME=device-metrics-exporter-mock docker-mock
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/docker'
building amd exporter mock container
Copying pre built amdsmi to docker
Note : user to include the built libs on to the container
'/usr/src/github.com/ROCm/device-metrics-exporter/assets/amd_smi_lib/x86_64/RHEL9/lib/libamd_smi.so.26' -> '/usr/src/github.com/ROCm/device-metrics-exporter/docker/libamd_smi.so.26'
'/usr/src/github.com/ROCm/device-metrics-exporter/assets/amd_smi_lib/x86_64/RHEL9/lib/libamd_smi.so.26.4.0' -> '/usr/src/github.com/ROCm/device-metrics-exporter/docker/libamd_smi.so.26.4.0'
Copying prebuilt rocprofiler libs and binary
'/usr/src/github.com/ROCm/device-metrics-exporter/assets/rocprofiler/librocpclient.so' -> '/usr/src/github.com/ROCm/device-metrics-exporter/docker/librocpclient.so'
'/usr/src/github.com/ROCm/device-metrics-exporter/assets/rocprofiler/rocpctl' -> '/usr/src/github.com/ROCm/device-metrics-exporter/docker/rocpctl'
Fetching amdgpu.ids from Mesa libdrm...
Appending unreleased GPU entries from assets/drm/amdgpu.ids.extra...
amdgpu.ids written to /usr/src/github.com/ROCm/device-metrics-exporter/docker/amdgpu.ids
[+] Building 4.7s (4/14) docker:default
=> [internal] load build definition from Dockerfile.exporter-mock-release 0.1s
[+] Building 6.1s (5/14) docker:default
[+] Building 9.7s (5/14) docker:default
=> [internal] load build definition from Dockerfile.exporter-mock-release 0.1s
[+] Building 41.3s (6/14) docker:default
[+] Building 42.8s (12/14) docker:default
[+] Building 44.5s (15/15) FINISHED docker:default
=> [internal] load build definition from Dockerfile.exporter-mock-release 0.1s
=> => transferring dockerfile: 1.46kB 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:22.04 2.4s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [ 1/10] FROM docker.io/library/ubuntu:22.04@sha256:4f838adc7181d9039ac795a7d0aba05a9bd9ecd480d294483169c5def983b64d 3.0s
=> => resolve docker.io/library/ubuntu:22.04@sha256:4f838adc7181d9039ac795a7d0aba05a9bd9ecd480d294483169c5def983b64d 0.1s
=> => sha256:4f838adc7181d9039ac795a7d0aba05a9bd9ecd480d294483169c5def983b64d 6.69kB / 6.69kB 0.0sa => => sha256:ce941a2a18bbb922e434d6d6d2b31e571a5c3826eaf6ada0a41dcc905bd2d906 424B / 424B 0.0s
=> => sha256:86f1a8d7b38e7a014c249cf2ca573c8ff7ce3cca128c5c06dcee758813726f90 2.05kB / 2.05kB 0.0s
=> => sha256:40d16f30db405106ef8074779bdf41f012465c2a785bbeaa2eab9f2081099b47 29.74MB / 29.74MB 2.0s
=> => extracting sha256:40d16f30db405106ef8074779bdf41f012465c2a785bbeaa2eab9f2081099b47 0.8s
=> [internal] load build context 1.6s
=> => transferring context: 271.65MB 1.6s
=> [ 2/10] RUN apt update && apt install -y sudo findutils procps file tar jq iproute2 ethtool libatomic1 jq vim curl && apt clean && rm -rf /va 35.1s
=> [ 3/10] RUN mkdir -p /home/amd/bin/ /home/amd/tools/ /mockdata 0.7s
=> [ 4/10] ADD ./rocpctl-mock /home/amd/bin/rocpctl 0.2s
=> [ 5/10] ADD ./gpuctl ./gpuagent ./metricsclient ./amdgpuhealth /home/amd/bin/ 0.2s => [ 6/10] ADD ./amd-metrics-exporter /home/amd/bin/server 0.2s
=> [ 7/10] ADD ./entrypoint.sh /home/amd/tools/entrypoint.sh 0.2s
=> [ 8/10] ADD ./LICENSE /licenses/LICENSE 0.2s
=> [ 9/10] RUN chmod +x /home/amd/tools/entrypoint.sh 0.5s
=> [10/10] COPY ./mockdata /mockdata 0.2s
=> exporting to image 1.2s
=> => exporting layers 1.2s
=> => writing image sha256:f28d2dc661f49081dba1d401fdb081302e075097364af8e89734708216b2d8d8 0.0s
=> => naming to docker.io/rocm/device-metrics-exporter-mock:latest 0.0s
1 warning found (use docker --debug to expand):
- UndefinedVar: Usage of undefined variable '$LD_LIBRARY_PATH' (line 13)
total 437592
drwxr-xr-x 1 paryan dialout 992 Jun 10 17:54 .
drwxr-xr-x 1 paryan dialout 1280 Jun 10 13:33 ..
-rw-r--r-- 1 paryan dialout 6148 Jun 1 21:58 .DS_Store
-rw-r--r-- 1 paryan dialout 2527 May 28 09:26 Dockerfile.ainic.exporter-release
-rw-r--r-- 1 paryan dialout 3790 May 28 09:26 Dockerfile.azure.linux3.exporter-release
-rw-r--r-- 1 paryan dialout 1402 May 28 09:26 Dockerfile.exporter-mock-release
-rw-r--r-- 1 paryan dialout 4240 Jun 10 11:55 Dockerfile.exporter-release
-rw-r--r-- 1 paryan dialout 3132 Jun 10 11:55 Dockerfile.sriov.exporter-release
-rw-r--r-- 1 paryan dialout 2147 Jun 10 11:55 Dockerfile.sriov.ub22.exporter-release
-rw-r--r-- 1 paryan dialout 11357 Jun 10 17:54 LICENSE
-rw-r--r-- 1 paryan dialout 4174 Jun 10 11:55 Makefile
-rwxr-xr-x 2 paryan dialout 81613043 Jun 10 17:54 amd-metrics-exporter
-rw-r--r-- 1 paryan dialout 25477 Jun 10 17:54 amdgpu.ids
-rwxr-xr-x 2 paryan dialout 72757553 Jun 10 17:54 amdgpuhealth
-rwxr-xr-x 1 paryan dialout 966 May 28 09:26 build_post_docker.sh
-rwxr-xr-x 1 paryan dialout 3825 Jun 10 11:55 build_prep_docker.sh
-rw-r--r-- 1 paryan dialout 158623453 Jun 10 17:45 device-metrics-exporter-mock-latest.tar.gz
-rw-r--r-- 1 paryan dialout 1138 May 28 09:26 entrypoint-host.sh
-rw-r--r-- 1 paryan dialout 1378 Jun 10 11:55 entrypoint.sh
-rwxr-xr-x 1 paryan dialout 31943088 Jun 4 04:49 gpuagent
-rwxr-xr-x 2 paryan dialout 14068024 Jun 10 11:55 gpuctl
-rw-r--r-- 1 paryan dialout 4260 Jun 10 11:55 install-rocm-tarball.sh
-rwxr-xr-x 1 paryan dialout 3205872 Jun 10 17:54 libamd_smi.so.26
-rwxr-xr-x 1 paryan dialout 3205872 Jun 10 17:54 libamd_smi.so.26.4.0
-rwxr-xr-x 1 paryan dialout 1901288 Jun 10 17:54 librocpclient.so
-rw-r--r-- 2 paryan dialout 6731 Jun 10 11:55 metrics-exporter-ts.sh
-rwxr-xr-x 2 paryan dialout 71063927 Jun 10 17:54 metricsclient
drwxr-xr-x 1 paryan dialout 160 Jun 1 21:58 mockdata
-rwxr-xr-x 1 paryan dialout 141456 Jun 10 17:54 rocpctl
-rwxr-xr-x 2 paryan dialout 16616 Jun 10 17:54 rocpctl-mock
drwxr-xr-x 1 paryan dialout 128 Jun 10 11:55 testrunner
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/docker'
make -C docker docker-save TOP_DIR=/usr/src/github.com/ROCm/device-metrics-exporter EXPORTER_IMAGE_NAME=device-metrics-exporter-mock
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/docker'
saving docker image to device-metrics-exporter-mock-latest.tar.gz
WARN[2026-06-10T17:55:22.569995862Z] healthcheck failed actualDuration="224.124µs" error="Unavailable: connection error: desc = \"transport: Error while dialing: only one connection allowed\"" timeout=15s
ERRO[2026-06-10T17:55:27.570818889Z] healthcheck failed fatally error="session healthcheck failed fatally: Unavailable: connection error: desc = \"transport: Error while dialing: only one connection allowed\""
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/docker'
make[1]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter'
make e2e-test
make[1]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter'
Makefile.package:162: warning: overriding recipe for target 'debpkg-sriov-clean'
Makefile.package:154: warning: ignoring old recipe for target 'debpkg-sriov-clean'
make -C test/e2e
make[2]: Entering directory '/usr/src/github.com/ROCm/device-metrics-exporter/test/e2e'
go clean -testcache && DOCKER_REGISTRY=docker.io/rocm EXPORTER_IMAGE_NAME=device-metrics-exporter-mock EXPORTER_IMAGE_TAG=latest go test -test.timeout=30m -clean-after-test -v "";
=== RUN Test
2026/06/10 17:56:25 Using exporter image: docker.io/rocm/device-metrics-exporter-mock:latest
2026/06/10 17:56:25 e2econfig : {ContainerName:test_exporter Mode:EXPORTER_MOCK ImageURL:docker.io/rocm/device-metrics-exporter-mock:latest}
2026/06/10 17:56:25 docker stop test_exporter
2026/06/10 17:56:25 local command out err exit status 1
2026/06/10 17:56:27 docker run --rm -itd --privileged --name test_exporter -p 5002:5002 -p 5000:5000 -v /usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/config_test/:/etc/metrics -e SIM_ENABLE=1 -e AMD_METRICS_EXPORTER_ENABLE_CPER=1 -e PATH=$PATH:/home/amd/bin/ docker.io/rocm/device-metrics-exporter-mock:latest
INFO[2026-06-10T17:56:28.883327372Z] connecting to shim 4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829 address="unix:///run/containerd/s/17e9ff577e96315a9de2d084d89e1614a1623ff185202eb9e579c55b9beed2e3" namespace=moby protocol=ttrpc version=3
time="2026-06-10T17:56:28.958022022Z" level=error msg="failed to enable controllers ([cpuset cpu io memory pids])" error="failed to write subtree controllers [cpuset cpu io memory pids] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory" runtime=io.containerd.runc.v2
time="2026-06-10T17:56:28.961509037Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829/memory.events\": no such file or directory"
INFO[2026-06-10T17:56:29.018169239Z] sbJoin: gwep4 ''->'d8676c582c46', gwep6 ''->'' eid=d8676c582c46 ep=test_exporter net=bridge nid=d5ac80d8091c
write config success : {}
2026/06/10 17:56:54 SetUpSuite done with config path :/usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/config_test/config.json
2026/06/10 17:56:54 s:2026-06-10 17:56:54.500309765 +0000 UTC m=+28.969361205 Validating Cluster
2026/06/10 17:56:54 SetUpTest Config file :
2026/06/10 17:56:54 Testing basic http response after docker deployment
2026/06/10 17:56:54 Test E2ESuite.Test001FirstDeplymentDefaults PASSED
2026/06/10 17:56:54 s:2026-06-10 17:56:54.840691345 +0000 UTC m=+29.309742785 Validating Cluster
2026/06/10 17:56:54 SetUpTest Config file :
2026/06/10 17:56:54 Testing non mandatatory label update
write config success : {
"GPUConfig": {
"Labels": [
"gpu_uuid"
]
}
}
2026/06/10 17:57:00 Test E2ESuite.Test002NonMandatoryLabelUpdate PASSED
2026/06/10 17:57:00 s:2026-06-10 17:57:00.093122117 +0000 UTC m=+34.562173557 Validating Cluster
2026/06/10 17:57:00 SetUpTest Config file : GPUConfig:{Labels:"gpu_uuid"}
2026/06/10 17:57:00 Testing KFD_PROCESS_ID label is optional and can be enabled via ConfigMap
write config success : {
"GPUConfig": {
"Labels": [
"kfd_process_id"
]
}
}
2026/06/10 17:57:05 Test E2ESuite.Test002bKFDProcessIdOptional PASSED
2026/06/10 17:57:05 s:2026-06-10 17:57:05.367489226 +0000 UTC m=+39.836540666 Validating Cluster
2026/06/10 17:57:05 SetUpTest Config file : GPUConfig:{Labels:"kfd_process_id"}
2026/06/10 17:57:05 test non mandatory invalid label update, should pick only valid labels
write config success : {
"GPUConfig": {
"Labels": [
"gpu_if",
"gpu_uuid"
]
}
}
2026/06/10 17:57:10 Test E2ESuite.Test003InvalidLabel PASSED
2026/06/10 17:57:10 s:2026-06-10 17:57:10.622848143 +0000 UTC m=+45.091899624 Validating Cluster
2026/06/10 17:57:10 SetUpTest Config file : GPUConfig:{Labels:"gpu_if" Labels:"gpu_uuid"}
2026/06/10 17:57:10 test non mandatory field update
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health",
"gpu_total_vram",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_if",
"gpu_uuid"
]
}
}
2026/06/10 17:57:15 Test E2ESuite.Test004FieldUpdate PASSED
2026/06/10 17:57:15 s:2026-06-10 17:57:15.796124798 +0000 UTC m=+50.265176280 Validating Cluster
2026/06/10 17:57:15 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_total_vram" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_if" Labels:"gpu_uuid"}
2026/06/10 17:57:15 test non mandatory invalid field update
write config success : {
"GPUConfig": {
"Fields": [
"invalid_config",
"gpu_health",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_if",
"gpu_uuid"
]
}
}
2026/06/10 17:57:20 Test E2ESuite.Test005InvalidFieldUpdate PASSED
2026/06/10 17:57:20 s:2026-06-10 17:57:20.970684819 +0000 UTC m=+55.439736259 Validating Cluster
2026/06/10 17:57:21 SetUpTest Config file : GPUConfig:{Fields:"invalid_config" Fields:"gpu_health" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_if" Labels:"gpu_uuid"}
2026/06/10 17:57:21 update server port
write config success : {
"ServerPort": 5002,
"GPUConfig": {
"Fields": [
"invalid_config",
"gpu_health",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_if",
"gpu_uuid"
]
}
}
2026/06/10 17:57:26 Test E2ESuite.Test006ServerPortUpdate PASSED
2026/06/10 17:57:26 s:2026-06-10 17:57:26.148160235 +0000 UTC m=+60.617211717 Validating Cluster
2026/06/10 17:57:26 SetUpTest Config file : ServerPort:5002 GPUConfig:{Fields:"invalid_config" Fields:"gpu_health" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_if" Labels:"gpu_uuid"}
2026/06/10 17:57:26 delete metric config should revert all configs and back to default
2026/06/10 17:57:31 Test E2ESuite.Test007DeleteConfig PASSED
2026/06/10 17:57:31 s:2026-06-10 17:57:31.402143621 +0000 UTC m=+65.871195061 Validating Cluster
2026/06/10 17:57:31 SetUpTest Config file :
2026/06/10 17:57:31 create config file after delete
write config success : {
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
]
}
}
2026/06/10 17:57:36 Test E2ESuite.Test008RecreateConfigFile PASSED
2026/06/10 17:57:36 s:2026-06-10 17:57:36.655585886 +0000 UTC m=+71.124637326 Validating Cluster
2026/06/10 17:57:36 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id"}
2026/06/10 17:57:36 update server port after config recreate
write config success : {
"ServerPort": 5002,
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
]
}
}
2026/06/10 17:57:46 Test E2ESuite.Test009ServerPortAfterRecreateConfig PASSED
2026/06/10 17:57:46 s:2026-06-10 17:57:46.894510793 +0000 UTC m=+81.363562233 Validating Cluster
2026/06/10 17:57:47 SetUpTest Config file : ServerPort:5002 GPUConfig:{Labels:"gpu_id" Labels:"job_id"}
2026/06/10 17:57:47 update server port with 0
write config success : {
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
]
}
}
2026/06/10 17:57:52 Test E2ESuite.Test010ServerInvalidPortUpdate PASSED
2026/06/10 17:57:52 s:2026-06-10 17:57:52.144064462 +0000 UTC m=+86.613115902 Validating Cluster
2026/06/10 17:57:52 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id"}
2026/06/10 17:57:52 creating new server server_noconfig
2026/06/10 17:57:52 cleaning up any old instances of same name server_noconfig
2026/06/10 17:57:52 docker stop server_noconfig
2026/06/10 17:57:52 local command out err exit status 1
2026/06/10 17:57:54 docker run --rm -itd --privileged --name server_noconfig -p 5003:5000 -e SIM_ENABLE=1 -e AMD_METRICS_EXPORTER_ENABLE_CPER=1 -e PATH=$PATH:/home/amd/bin/ docker.io/rocm/device-metrics-exporter-mock:latest
INFO[2026-06-10T17:57:55.001951512Z] connecting to shim 2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097 address="unix:///run/containerd/s/b8948e7a1e9ced3f7904c9228043b74ce4e3a273df22fda8a60cb1b570aef6cd" namespace=moby protocol=ttrpc version=3
time="2026-06-10T17:57:55.071318285Z" level=error msg="failed to enable controllers ([cpuset cpu io memory pids])" error="failed to write subtree controllers [cpuset cpu io memory pids] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory" runtime=io.containerd.runc.v2
time="2026-06-10T17:57:55.078407399Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097/memory.events\": no such file or directory"
INFO[2026-06-10T17:57:55.143482495Z] sbJoin: gwep4 ''->'e5d575947674', gwep6 ''->'' eid=e5d575947674 ep=server_noconfig net=bridge nid=d5ac80d8091c
2026/06/10 17:57:55 waiting for container server_noconfig to start
2026/06/10 17:58:20 creation of new container server_noconfig done
2026/06/10 17:58:20 deleting container server_noconfig
2026/06/10 17:58:20 docker stop server_noconfig
INFO[2026-06-10T17:58:20.733681793Z] received task-delete event from containerd container=2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
INFO[2026-06-10T17:58:20.736678895Z] shim disconnected id=2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097 namespace=moby
INFO[2026-06-10T17:58:20.736994560Z] cleaning up after shim disconnected id=2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097 namespace=moby
INFO[2026-06-10T17:58:20.737179142Z] cleaning up dead shim id=2df8a089a742dc8dc39ca35c44e7526b9b5cd7a7c72d496b35d95e64774b9097 namespace=moby
2026/06/10 17:58:21 Test E2ESuite.Test011ContainerWithoutConfig PASSED
2026/06/10 17:58:21 s:2026-06-10 17:58:21.101878665 +0000 UTC m=+115.570930105 Validating Cluster
2026/06/10 17:58:21 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id"}
2026/06/10 17:58:21 Testing custom label update
write config success : {
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"cLabel1": "cValue1",
"cLabel2": "cValue2"
}
}
}
2026/06/10 17:58:26 Test E2ESuite.Test012CustomLabelUpdate PASSED
2026/06/10 17:58:26 s:2026-06-10 17:58:26.372675923 +0000 UTC m=+120.841727363 Validating Cluster
2026/06/10 17:58:26 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"cLabel1" value:"cValue1"} CustomLabels:{key:"cLabel2" value:"cValue2"}}
2026/06/10 17:58:26 Testing mandatory labels supplied as custom labels
write config success : {
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"cLabel1": "cValue1",
"card_model": "custom_card_model",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
2026/06/10 17:58:31 Test E2ESuite.Test013MandatoryLabelsAsCustomLabels PASSED
2026/06/10 17:58:31 s:2026-06-10 17:58:31.634817953 +0000 UTC m=+126.103869393 Validating Cluster
2026/06/10 17:58:31 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"cLabel1" value:"cValue1"} CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}}
2026/06/10 17:58:31 Testing existing labels supplied as custom labels
write config success : {
"GPUConfig": {
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
2026/06/10 17:58:36 Test E2ESuite.Test014ExistingLabelsAsCustomLabels PASSED
2026/06/10 17:58:36 s:2026-06-10 17:58:36.88829009 +0000 UTC m=+131.357341530 Validating Cluster
2026/06/10 17:58:37 SetUpTest Config file : GPUConfig:{Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}}
2026/06/10 17:58:37 test prefix update
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health",
"gpu_total_vram",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health",
"gpu_total_vram",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
},
"CommonConfig": {
"MetricsFieldPrefix": "amd_"
}
}
2026/06/10 17:58:42 Prefix Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_total_vram" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{MetricsFieldPrefix:"amd_"}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health",
"gpu_total_vram",
"gpu_ecc_uncorrect_gfx",
"gpu_umc_activity",
"gpu_mma_activity"
],
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
},
"CommonConfig": {}
}
2026/06/10 17:58:47 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_total_vram" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
2026/06/10 17:58:47 Prefix Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_total_vram" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
2026/06/10 17:58:47 Test E2ESuite.Test015FieldPrefixUpdate PASSED
2026/06/10 17:58:47 s:2026-06-10 17:58:47.091214346 +0000 UTC m=+141.560265786 Validating Cluster
2026/06/10 17:58:47 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_total_vram" Fields:"gpu_ecc_uncorrect_gfx" Fields:"gpu_umc_activity" Fields:"gpu_mma_activity" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
2026/06/10 17:58:47 Health Service Reconnect
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health"
],
"Labels": [
"gpu_id",
"job_id"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
},
"CommonConfig": {}
}
2026/06/10 17:58:52 Prefix Config file : GPUConfig:{Fields:"gpu_health" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
2026/06/10 17:58:52 all gpu in expected health state [1]
2026/06/10 17:58:52 executing cmd [docker exec -it test_exporter pkill gpuagent]
2026/06/10 17:58:52 local command out err exit status 1
2026/06/10 17:58:57 gpu response : # HELP gpu_health Health of the GPU (0 = Unhealthy | 1 = Healthy)
# TYPE gpu_health gauge
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 17:58:57 executing cmd [docker exec -it test_exporter gpuagent &]
2026/06/10 17:59:02 gpu response : # HELP gpu_health Health of the GPU (0 = Unhealthy | 1 = Healthy)
# TYPE gpu_health gauge
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
gpu_health{card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number"} 1
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 17:59:02 all gpu in expected health state [1]
2026/06/10 17:59:02 Test E2ESuite.Test016HealthSvcReconnect PASSED
2026/06/10 17:59:02 s:2026-06-10 17:59:02.634872461 +0000 UTC m=+157.103923901 Validating Cluster
2026/06/10 17:59:02 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Labels:"gpu_id" Labels:"job_id" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
},
"CommonConfig": {}
}
2026/06/10 17:59:07 docker cp slurm_job.json test_exporter:/var/run/exporter/3
2026/06/10 17:59:12 GPU["0"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["1"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["2"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["3"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["4"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["5"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["6"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 GPU["7"] field[gpu_health] has correct job labels
2026/06/10 17:59:12 Job labels verified successfully: present on GPUs 0-7
2026/06/10 17:59:12 Test E2ESuite.Test017SlurmWorkloadSim PASSED
2026/06/10 17:59:12 s:2026-06-10 17:59:12.979921571 +0000 UTC m=+167.448973053 Validating Cluster
2026/06/10 17:59:13 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}} CommonConfig:{}
2026/06/10 17:59:13 Disabling health service via SetCommonConfigHealth(false)
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
},
"CommonConfig": {
"HealthService": {}
}
}
2026/06/10 17:59:18 local command out err exit status 1
2026/06/10 17:59:18
2026/06/10 17:59:18 Removing commonconfig and verifying metricsclient --json returns non-empty output
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
2026/06/10 17:59:23 {"GPUState":[{"ID":"13","UUID":"0002ac11-420d-4242-f349-5610e03b3345","Health":"healthy"},{"ID":"2","UUID":"0002ac11-4202-4242-4e9e-e81357f56712","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"5","UUID":"0002ac11-4205-4242-f732-6218b484bc3c","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"14","UUID":"0002ac11-420e-4242-c5e0-1807e708fd12","Health":"healthy"},{"ID":"10","UUID":"0002ac11-420a-4242-266f-e82b30511920","Health":"healthy"},{"ID":"8","UUID":"0002ac11-4208-4242-8c5c-5466f9a4267f","Health":"healthy"},{"ID":"15","UUID":"0002ac11-420f-4242-d092-a83786089721","Health":"healthy"},{"ID":"7","UUID":"0002ac11-4207-4242-1ef3-5f5e40376a3c","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"6","UUID":"0002ac11-4206-4242-dff8-4d663aa8b13c","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"12","UUID":"0002ac11-420c-4242-7098-165a849a863b","Health":"healthy"},{"ID":"11","UUID":"0002ac11-420b-4242-cbb9-2a1d4c4f8423","Health":"healthy"},{"ID":"4","UUID":"0002ac11-4204-4242-45a0-b32436919969","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"9","UUID":"0002ac11-4209-4242-7bd3-2a1e2cf3f16e","Health":"healthy"},{"ID":"3","UUID":"0002ac11-4203-4242-0afb-e72eef18661b","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"0","UUID":"0002ac11-4200-4242-37c1-813fe989ab24","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]},{"ID":"1","UUID":"0002ac11-4201-4242-a17e-3a2163821b1e","Health":"healthy","AssociatedWorkload":["Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11"]}]}
2026/06/10 17:59:23 Test E2ESuite.Test018HealthSvcToggle PASSED
2026/06/10 17:59:23 s:2026-06-10 17:59:23.731558266 +0000 UTC m=+178.200609706 Validating Cluster
2026/06/10 17:59:23 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}}
2026/06/10 17:59:23 Testing ECC error injection via metricsclient
2026/06/10 17:59:23 docker cp ecc_errors.json test_exporter:/tmp/ecc_errors.json
2026/06/10 17:59:24 ECC injection output: {"ID":"0","Fields":["GPU_ECC_UNCORRECT_FUSE","GPU_ECC_UNCORRECT_DF"]}
ID UUID Health Associated Workload
------------------------------------------------
0 0002ac11-4200-4242-37c1-813fe989ab24 healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
1 0002ac11-4201-4242-a17e-3a2163821b1e healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
2 0002ac11-4202-4242-4e9e-e81357f56712 healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
3 0002ac11-4203-4242-0afb-e72eef18661b healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
4 0002ac11-4204-4242-45a0-b32436919969 healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
5 0002ac11-4205-4242-f732-6218b484bc3c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
6 0002ac11-4206-4242-dff8-4d663aa8b13c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
7 0002ac11-4207-4242-1ef3-5f5e40376a3c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
8 0002ac11-4208-4242-8c5c-5466f9a4267f healthy []
9 0002ac11-4209-4242-7bd3-2a1e2cf3f16e healthy []
10 0002ac11-420a-4242-266f-e82b30511920 healthy []
11 0002ac11-420b-4242-cbb9-2a1d4c4f8423 healthy []
12 0002ac11-420c-4242-7098-165a849a863b healthy []
13 0002ac11-420d-4242-f349-5610e03b3345 healthy []
14 0002ac11-420e-4242-c5e0-1807e708fd12 healthy []
15 0002ac11-420f-4242-d092-a83786089721 healthy []
------------------------------------------------
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_uncorrect_fuse",
"gpu_ecc_uncorrect_df",
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_uncorrect_fuse",
"gpu_ecc_uncorrect_df",
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
2026/06/10 17:59:29 all gpu in expected health state [0]
2026/06/10 17:59:29 ECC health state verified successfully for GPU 0
2026/06/10 17:59:29 docker cp clear_ecc_errors.json test_exporter:/tmp/clear_ecc_errors.json
2026/06/10 17:59:29 ECC clear output: {"ID":"0","Fields":["GPU_ECC_UNCORRECT_FUSE","GPU_ECC_UNCORRECT_DF"]}
ID UUID Health Associated Workload
------------------------------------------------
0 0002ac11-4200-4242-37c1-813fe989ab24 unhealthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
1 0002ac11-4201-4242-a17e-3a2163821b1e healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
2 0002ac11-4202-4242-4e9e-e81357f56712 healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
3 0002ac11-4203-4242-0afb-e72eef18661b healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
4 0002ac11-4204-4242-45a0-b32436919969 healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
5 0002ac11-4205-4242-f732-6218b484bc3c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
6 0002ac11-4206-4242-dff8-4d663aa8b13c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
7 0002ac11-4207-4242-1ef3-5f5e40376a3c healthy [Job: 742, User: user_7kq, Partition: 256C8G1H_MI325X_Ubuntu22, Cluster: aac11]
8 0002ac11-4208-4242-8c5c-5466f9a4267f healthy []
9 0002ac11-4209-4242-7bd3-2a1e2cf3f16e healthy []
10 0002ac11-420a-4242-266f-e82b30511920 healthy []
11 0002ac11-420b-4242-cbb9-2a1d4c4f8423 healthy []
12 0002ac11-420c-4242-7098-165a849a863b healthy []
13 0002ac11-420d-4242-f349-5610e03b3345 healthy []
14 0002ac11-420e-4242-c5e0-1807e708fd12 healthy []
15 0002ac11-420f-4242-d092-a83786089721 healthy []
------------------------------------------------
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_uncorrect_fuse",
"gpu_ecc_uncorrect_df",
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_uncorrect_fuse",
"gpu_ecc_uncorrect_df",
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
}
}
}
2026/06/10 17:59:34 all gpu in expected health state [1]
2026/06/10 17:59:34 ECC health state successfully cleared for GPU 0
2026/06/10 17:59:34 Test E2ESuite.Test019ECCErrorInjection PASSED
2026/06/10 17:59:34 s:2026-06-10 17:59:34.832606431 +0000 UTC m=+189.301657871 Validating Cluster
2026/06/10 17:59:34 SetUpTest Config file : GPUConfig:{Fields:"gpu_ecc_uncorrect_fuse" Fields:"gpu_ecc_uncorrect_df" Fields:"gpu_health" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"}}
2026/06/10 17:59:34 Testing profiler failure handling
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_uncorrect_fuse",
"gpu_ecc_uncorrect_df",
"gpu_health"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": true
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_prof_sm_active"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": true
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_prof_sm_active"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
}
}
2026/06/10 17:59:45 String 'rocpclient has been disabled after system failure' found in exporter log
2026/06/10 17:59:45 Test E2ESuite.Test020ProfilerFailureHandling PASSED
2026/06/10 17:59:45 s:2026-06-10 17:59:45.166849538 +0000 UTC m=+199.635900978 Validating Cluster
2026/06/10 17:59:45 SetUpTest Config file : GPUConfig:{Fields:"gpu_prof_sm_active" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}}
2026/06/10 17:59:45 Testing enabling gpu_afid_errors field in config
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
}
}
2026/06/10 17:59:45 docker exec test_exporter rm -rf /mockdata/inband-ras/error_list
2026/06/10 17:59:50 Test E2ESuite.Test021EnableGpuAfidErrorsField PASSED
2026/06/10 17:59:50 s:2026-06-10 17:59:50.520835341 +0000 UTC m=+204.989886781 Validating Cluster
2026/06/10 17:59:50 SetUpTest Config file : GPUConfig:{Fields:"gpu_afid_errors" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}}
2026/06/10 17:59:50 Testing logger config update
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 17:59:55 String 'starting server on' found in exporter log
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "DEBUG",
"MaxFileSizeMB": 2,
"MaxBackups": 1,
"MaxAgeDays": 2
}
}
}
2026/06/10 18:00:01 String 'loading new config on' found in exporter log
2026/06/10 18:00:01 Test E2ESuite.Test022LoggerConfigUpdate PASSED
2026/06/10 18:00:01 s:2026-06-10 18:00:01.015330166 +0000 UTC m=+215.484381648 Validating Cluster
2026/06/10 18:00:01 SetUpTest Config file : GPUConfig:{Fields:"gpu_afid_errors" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"DEBUG" MaxFileSizeMB:2 MaxBackups:1 MaxAgeDays:2}}
2026/06/10 18:00:01 Testing logger config with invalid log level
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INVALID_LEVEL",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:00:11 Test E2ESuite.Test023LoggerConfigWithInvalidLevel PASSED
2026/06/10 18:00:11 s:2026-06-10 18:00:11.221617147 +0000 UTC m=+225.690668629 Validating Cluster
2026/06/10 18:00:11 SetUpTest Config file : GPUConfig:{Fields:"gpu_afid_errors" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"INFO" MaxFileSizeMB:1 MaxBackups:1 MaxAgeDays:1}}
2026/06/10 18:00:11 Testing mock inband RAS error with AFID 35
write config success : {
"GPUConfig": {
"Fields": [
"gpu_afid_errors"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:00:16 docker exec test_exporter metricsclient -setup-mock-inbandras
2026/06/10 18:00:16 local command out err exit status 1
2026/06/10 18:00:16 docker exec test_exporter jq '.cper |= map(.afid = [35])' /mockdata/inband-ras/error_list > tmp.json
2026/06/10 18:00:16 local command out err exit status 2
2026/06/10 18:00:16 docker cp tmp.json test_exporter:/mockdata/inband-ras/error_list
2026/06/10 18:00:17 Updated mock inband RAS error file with AFID 35:
2026/06/10 18:00:19 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:20 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:21 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:22 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:23 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:24 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:25 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:26 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:27 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:28 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:29 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:30 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:31 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:32 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:33 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:34 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:35 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:36 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:37 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:38 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:39 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:40 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:41 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:42 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:43 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:44 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:45 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:46 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:47 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:48 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:49 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:50 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:51 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:52 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:53 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:54 Successfully verified AFID 35 in mock inband RAS error response
2026/06/10 18:00:54 docker exec test_exporter rm -rf /mockdata/inband-ras/error_list tmp.json
2026/06/10 18:00:54 Metrics response: # HELP gpu_afid_errors Last Occurred RAS Event associated AMD Field Identifier list
# TYPE gpu_afid_errors gauge
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="0",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="1",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="10",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="11",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="12",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="13",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="14",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="15",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="2",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="3",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="4",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="5",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="6",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="7",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="742",job_partition="256C8G1H_MI325X_Ubuntu22",job_user="user_7kq",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="8",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
gpu_afid_errors{afid_index="0",card_model="custom_card_model",card_vendor="cValue2",cluster_name="cValue1",container="",deployment_mode="baremetal",driver_version="cValue3",gpu_compute_partition_type="spx",gpu_id="9",gpu_memory_partition_type="nps1",gpu_partition_id="0",hostname="4cbc4d579647",job_id="",job_partition="",job_user="",namespace="",pod="",serial_number="custom_serial_number",severity="fatal"} 30
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
2026/06/10 18:00:54 Test E2ESuite.Test024MockInbandRAS FAILED
----------------------------------------------------------------------
FAIL: exporter_test.go:933: E2ESuite.Test024MockInbandRAS
exporter_test.go:945:
assert.Nil(c, err)
/usr/src/github.com/ROCm/device-metrics-exporter/vendor/github.com/stretchr/testify/assert/assertions.go:387:
t.Errorf("\n%s", ""+labeledOutput(content...))
... Error:
Error Trace: /usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/exporter_test.go:945
Error: Expected nil, but got: &errors.errorString{s:"empty response"}
exporter_test.go:964:
assert.Eventually(c, func() bool {
response, _ = s.getExporterResponse()
if response == "" {
log.Print("Waiting for metrics response to be available...")
return false
}
// Verify the response contains AFID 35
log.Printf("Metrics response: %s", response)
// Parse the response to get the field value of gpu_afid_errors
allgpus, err := testutils.ParsePrometheusMetrics(response)
if err != nil {
log.Printf("Failed to parse metrics: %v", err)
return false
}
// Check for AFID 35 in gpu_afid_errors field
foundAFID35 := false
for gpuId, gpu := range allgpus {
afidField, ok := gpu.Fields["gpu_afid_errors"]
if !ok {
continue
}
// Check if afid_index label contains "35"
if afidField.Value == "35" {
foundAFID35 = true
log.Printf("Found AFID 35 in gpu_afid_errors field for GPU[%v]", gpuId)
break
}
}
return foundAFID35
}, 35*time.Second, 1*time.Second)
/usr/src/github.com/ROCm/device-metrics-exporter/vendor/github.com/stretchr/testify/assert/assertions.go:387:
t.Errorf("\n%s", ""+labeledOutput(content...))
... Error:
Error Trace: /usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/exporter_test.go:964
Error: Condition never satisfied
2026/06/10 18:00:54 s:2026-06-10 18:00:54.235662415 +0000 UTC m=+268.704713855 Validating Cluster
2026/06/10 18:00:54 SetUpTest Config file : GPUConfig:{Fields:"gpu_afid_errors" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"INFO" MaxFileSizeMB:1 MaxBackups:1 MaxAgeDays:1}}
2026/06/10 18:00:54 Testing ECC deferred error metrics configuration
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_sdma",
"gpu_ecc_deferred_gfx",
"gpu_ecc_deferred_mmhub",
"gpu_ecc_deferred_athub",
"gpu_ecc_deferred_bif",
"gpu_ecc_deferred_hdp",
"gpu_ecc_deferred_xgmi_wafl",
"gpu_ecc_deferred_df",
"gpu_ecc_deferred_smn",
"gpu_ecc_deferred_sem",
"gpu_ecc_deferred_mp0",
"gpu_ecc_deferred_mp1",
"gpu_ecc_deferred_fuse",
"gpu_ecc_deferred_umc",
"gpu_ecc_deferred_mca",
"gpu_ecc_deferred_vcn",
"gpu_ecc_deferred_jpeg",
"gpu_ecc_deferred_ih",
"gpu_ecc_deferred_mpio"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:00:59 GPU["5"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["6"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["11"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["12"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["13"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["4"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["0"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["15"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["2"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["1"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["10"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["14"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["3"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["8"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["9"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 GPU["7"]: Found gpu_ecc_deferred_total metric
2026/06/10 18:00:59 Successfully validated ECC deferred error metrics configuration
2026/06/10 18:00:59 Test E2ESuite.Test025ECCDeferredErrorConfig PASSED
2026/06/10 18:00:59 s:2026-06-10 18:00:59.422020174 +0000 UTC m=+273.891071614 Validating Cluster
2026/06/10 18:00:59 SetUpTest Config file : GPUConfig:{Fields:"gpu_ecc_deferred_total" Fields:"gpu_ecc_deferred_sdma" Fields:"gpu_ecc_deferred_gfx" Fields:"gpu_ecc_deferred_mmhub" Fields:"gpu_ecc_deferred_athub" Fields:"gpu_ecc_deferred_bif" Fields:"gpu_ecc_deferred_hdp" Fields:"gpu_ecc_deferred_xgmi_wafl" Fields:"gpu_ecc_deferred_df" Fields:"gpu_ecc_deferred_smn" Fields:"gpu_ecc_deferred_sem" Fields:"gpu_ecc_deferred_mp0" Fields:"gpu_ecc_deferred_mp1" Fields:"gpu_ecc_deferred_fuse" Fields:"gpu_ecc_deferred_umc" Fields:"gpu_ecc_deferred_mca" Fields:"gpu_ecc_deferred_vcn" Fields:"gpu_ecc_deferred_jpeg" Fields:"gpu_ecc_deferred_ih" Fields:"gpu_ecc_deferred_mpio" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"INFO" MaxFileSizeMB:1 MaxBackups:1 MaxAgeDays:1}}
2026/06/10 18:00:59 Testing ECC deferred error metrics can be disabled
write config success : {
"GPUConfig": {
"Fields": [
"gpu_health",
"gpu_package_power",
"gpu_total_vram"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:01:04 Successfully validated ECC deferred error metrics are disabled
2026/06/10 18:01:04 Test E2ESuite.Test026ECCDeferredErrorDisableConfig PASSED
2026/06/10 18:01:04 s:2026-06-10 18:01:04.598284218 +0000 UTC m=+279.067335700 Validating Cluster
2026/06/10 18:01:04 SetUpTest Config file : GPUConfig:{Fields:"gpu_health" Fields:"gpu_package_power" Fields:"gpu_total_vram" Labels:"job_id" Labels:"job_partition" Labels:"job_user" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"INFO" MaxFileSizeMB:1 MaxBackups:1 MaxAgeDays:1}}
2026/06/10 18:01:04 Testing ECC deferred error metrics have correct labels
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_umc"
],
"Labels": [
"job_id",
"job_partition",
"job_user"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_umc"
],
"Labels": [
"gpu_id",
"gpu_uuid",
"hostname"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:01:09 GPU["14"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["14"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["2"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["2"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["7"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["7"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["13"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["13"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["15"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["15"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["3"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["3"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["8"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["8"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["9"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["9"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["10"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["10"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["4"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["4"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["5"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["5"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["6"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["6"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["0"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["0"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["1"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["1"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["11"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["11"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 GPU["12"]: gpu_ecc_deferred_total has all expected labels
2026/06/10 18:01:09 GPU["12"]: gpu_ecc_deferred_umc has all expected labels
2026/06/10 18:01:09 Successfully validated ECC deferred error metrics labels
2026/06/10 18:01:09 Test E2ESuite.Test027ECCDeferredErrorLabels PASSED
2026/06/10 18:01:09 s:2026-06-10 18:01:09.781059213 +0000 UTC m=+284.250110653 Validating Cluster
2026/06/10 18:01:09 SetUpTest Config file : GPUConfig:{Fields:"gpu_ecc_deferred_total" Fields:"gpu_ecc_deferred_umc" Labels:"gpu_id" Labels:"gpu_uuid" Labels:"hostname" CustomLabels:{key:"card_model" value:"custom_card_model"} CustomLabels:{key:"card_vendor" value:"cValue2"} CustomLabels:{key:"cluster_name" value:"cValue1"} CustomLabels:{key:"driver_version" value:"cValue3"} CustomLabels:{key:"gpu_id" value:"custom_gpu_id"} CustomLabels:{key:"serial_number" value:"custom_serial_number"} ProfilerMetrics:{key:"all" value:false}} CommonConfig:{Logging:{Level:"INFO" MaxFileSizeMB:1 MaxBackups:1 MaxAgeDays:1}}
2026/06/10 18:01:09 Testing --exit-on-rocpctl-error: DME exits after rocpctl crash
2026/06/10 18:01:09 docker stop test_exporter
INFO[2026-06-10T18:01:10.109098260Z] received task-delete event from containerd container=4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
INFO[2026-06-10T18:01:10.114756218Z] shim disconnected id=4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829 namespace=moby
INFO[2026-06-10T18:01:10.116921160Z] cleaning up after shim disconnected id=4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829 namespace=moby
INFO[2026-06-10T18:01:10.116930785Z] cleaning up dead shim id=4cbc4d5796470d0cc028b210c360befdc2af0593143296ff943ec00359c38829 namespace=moby
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_umc"
],
"Labels": [
"gpu_id",
"gpu_uuid",
"hostname"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:01:12 docker run --rm -itd --privileged --name test_exporter -p 5000:5000 -p 5002:5002 -v /usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/config_test/:/etc/metrics -e SIM_ENABLE=1 -e AMD_METRICS_EXPORTER_ENABLE_CPER=1 -e PATH=$PATH:/home/amd/bin/ docker.io/rocm/device-metrics-exporter-mock:latest --exit-on-rocpctl-error=true
INFO[2026-06-10T18:01:13.206772388Z] connecting to shim c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd address="unix:///run/containerd/s/b559eed7ea469986a6d8ac46e5c7e1686622b8a69c010af8d1b83ef24a27bfaa" namespace=moby protocol=ttrpc version=3
time="2026-06-10T18:01:13.281476747Z" level=error msg="failed to enable controllers ([cpuset cpu io memory pids])" error="failed to write subtree controllers [cpuset cpu io memory pids] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory" runtime=io.containerd.runc.v2
time="2026-06-10T18:01:13.285503591Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd/memory.events\": no such file or directory"
INFO[2026-06-10T18:01:13.342020835Z] sbJoin: gwep4 ''->'b8b5d480d5d3', gwep6 ''->'' eid=b8b5d480d5d3 ep=test_exporter net=bridge nid=d5ac80d8091c
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_umc"
],
"Labels": [
"gpu_id",
"gpu_uuid",
"hostname"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": true
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
INFO[2026-06-10T18:01:54.058457622Z] received task-delete event from containerd container=c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
INFO[2026-06-10T18:01:54.059840320Z] shim disconnected id=c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd namespace=moby
INFO[2026-06-10T18:01:54.059904611Z] cleaning up after shim disconnected id=c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd namespace=moby
INFO[2026-06-10T18:01:54.059910111Z] cleaning up dead shim id=c2a2a23de82468e573607a04a48ca5e9b56c8341a0df78b224a8e7d541e320fd namespace=moby
write config success : {
"GPUConfig": {
"Fields": [
"gpu_ecc_deferred_total",
"gpu_ecc_deferred_umc"
],
"Labels": [
"gpu_id",
"gpu_uuid",
"hostname"
],
"CustomLabels": {
"card_model": "custom_card_model",
"card_vendor": "cValue2",
"cluster_name": "cValue1",
"driver_version": "cValue3",
"gpu_id": "custom_gpu_id",
"serial_number": "custom_serial_number"
},
"ProfilerMetrics": {
"all": false
}
},
"CommonConfig": {
"Logging": {
"Level": "INFO",
"MaxFileSizeMB": 1,
"MaxBackups": 1,
"MaxAgeDays": 1
}
}
}
2026/06/10 18:01:55 docker run --rm -itd --privileged --name test_exporter -p 5000:5000 -p 5002:5002 -v /usr/src/github.com/ROCm/device-metrics-exporter/test/e2e/config_test/:/etc/metrics -e SIM_ENABLE=1 -e AMD_METRICS_EXPORTER_ENABLE_CPER=1 -e PATH=$PATH:/home/amd/bin/ docker.io/rocm/device-metrics-exporter-mock:latest
INFO[2026-06-10T18:01:55.828679218Z] connecting to shim 23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e address="unix:///run/containerd/s/76396e7cf0feb4a47b8df739a27ef39b9e51e3e80a790a1abd43fd3175b56a32" namespace=moby protocol=ttrpc version=3
time="2026-06-10T18:01:55.904569859Z" level=error msg="failed to enable controllers ([cpuset cpu io memory pids])" error="failed to write subtree controllers [cpuset cpu io memory pids] to \"/sys/fs/cgroup/docker/cgroup.subtree_control\": write /sys/fs/cgroup/docker/cgroup.subtree_control: no such file or directory" runtime=io.containerd.runc.v2
time="2026-06-10T18:01:55.907117548Z" level=warning msg="error from *cgroupsv2.Manager.EventChan" error="failed to add inotify watch for \"/sys/fs/cgroup/docker/23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e/memory.events\": no such file or directory"
INFO[2026-06-10T18:01:55.961126395Z] sbJoin: gwep4 ''->'4d77204f4c73', gwep6 ''->'' eid=4d77204f4c73 ep=test_exporter net=bridge nid=d5ac80d8091c
2026/06/10 18:02:21 Test E2ESuite.Test028ExitOnRocpctlError PASSED
2026/06/10 18:02:26 s:2026-06-10 18:02:26.43568485 +0000 UTC m=+360.904736290 Validating Cluster
2026/06/10 18:02:26 cleaning setup after test
INFO[2026-06-10T18:02:26.763448149Z] shim disconnected id=23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e namespace=moby
INFO[2026-06-10T18:02:26.763499774Z] cleaning up after shim disconnected id=23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e namespace=moby
INFO[2026-06-10T18:02:26.763509607Z] cleaning up dead shim id=23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e namespace=moby
INFO[2026-06-10T18:02:26.763689772Z] received task-delete event from containerd container=23d0b07d42cb7312317392af71f05c7e211f9b78b1198a39c98133a68f39042e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
================================================================================
TEST SUMMARY
================================================================================
TEST NAME | RESULT
--------------------------------------------------------------------------------
E2ESuite.Test001FirstDeplymentDefaults | ✓ PASS
E2ESuite.Test002NonMandatoryLabelUpdate | ✓ PASS
E2ESuite.Test002bKFDProcessIdOptional | ✓ PASS
E2ESuite.Test003InvalidLabel | ✓ PASS
E2ESuite.Test004FieldUpdate | ✓ PASS
E2ESuite.Test005InvalidFieldUpdate | ✓ PASS
E2ESuite.Test006ServerPortUpdate | ✓ PASS
E2ESuite.Test007DeleteConfig | ✓ PASS
E2ESuite.Test008RecreateConfigFile | ✓ PASS
E2ESuite.Test009ServerPortAfterRecreateConfig | ✓ PASS
E2ESuite.Test010ServerInvalidPortUpdate | ✓ PASS
E2ESuite.Test011ContainerWithoutConfig | ✓ PASS
E2ESuite.Test012CustomLabelUpdate | ✓ PASS
E2ESuite.Test013MandatoryLabelsAsCustomLabels | ✓ PASS
E2ESuite.Test014ExistingLabelsAsCustomLabels | ✓ PASS
E2ESuite.Test015FieldPrefixUpdate | ✓ PASS
E2ESuite.Test016HealthSvcReconnect | ✓ PASS
E2ESuite.Test017SlurmWorkloadSim | ✓ PASS
E2ESuite.Test018HealthSvcToggle | ✓ PASS
E2ESuite.Test019ECCErrorInjection | ✓ PASS
E2ESuite.Test020ProfilerFailureHandling | ✓ PASS
E2ESuite.Test021EnableGpuAfidErrorsField | ✓ PASS
E2ESuite.Test022LoggerConfigUpdate | ✓ PASS
E2ESuite.Test023LoggerConfigWithInvalidLevel | ✓ PASS
E2ESuite.Test024MockInbandRAS | ✗ FAIL
E2ESuite.Test025ECCDeferredErrorConfig | ✓ PASS
E2ESuite.Test026ECCDeferredErrorDisableConfig | ✓ PASS
E2ESuite.Test027ECCDeferredErrorLabels | ✓ PASS
E2ESuite.Test028ExitOnRocpctlError | ✓ PASS
================================================================================
TOTAL: 29 | PASSED: 28 | SKIPPED: 0 | FAILED: 1
================================================================================
OOPS: 28 passed, 1 FAILED
--- FAIL: Test (363.68s)
FAIL
exit status 1
FAIL github.com/ROCm/device-metrics-exporter/test/e2e 363.813s
make[2]: *** [Makefile:7: all] Error 1
make[2]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter/test/e2e'
make[1]: *** [Makefile:515: e2e-test] Error 2
make[1]: Leaving directory '/usr/src/github.com/ROCm/device-metrics-exporter'
make: *** [Makefile:520: e2e] Error 2
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$
paryan@611e12e7a68f:/usr/src/github.com/ROCm/device-metrics-exporter$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment