Skip to content

Instantly share code, notes, and snippets.

@therevoman
Created July 26, 2021 20:59
Show Gist options
  • Save therevoman/c32fe63def347d7df4a949457c423f30 to your computer and use it in GitHub Desktop.
Save therevoman/c32fe63def347d7df4a949457c423f30 to your computer and use it in GitHub Desktop.
While attempting an upgrade of a single node cluster created using the assisted installer from candidate-4.8.0 to candidate-4.8.2 everything completed except the machine-config operator. The error looked like:
``` Master: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node sno-assisted is reporting: \"unexpected on-disk state validating against rendered-master-9adaa63f411b981cfffd0aba37e17729: expected target osImageURL \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce\\\", have \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:369c15034dee19e19aa725de19bfb91c49db28b1c57958104a278e7237eb1d3f\\\"\""```
Following the comment at the bottom of this Red Hat solution article https://access.redhat.com/solutions/5510891
I logged into the node via ssh and used the "expected target osImageUrl" in the following command
`sudo /run/bin/machine-config-daemon pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce`
```[root@sno-assisted ~]# /run/bin/machine-config-daemon pivot quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce
I0726 20:47:32.671630 76863 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-521469848 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce
I0726 20:48:07.534743 76863 rpm-ostree.go:278] Running captured: rpm-ostree status --json
I0726 20:48:07.595925 76863 rpm-ostree.go:186] Previous pivot: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:369c15034dee19e19aa725de19bfb91c49db28b1c57958104a278e7237eb1d3f
I0726 20:48:09.001313 76863 rpm-ostree.go:218] Pivoting to: 48.84.202107202156-0 (57356ce06823a842bbb83e2ec652f7003f373d421589a0d37fa842ba918b170d)
I0726 20:48:09.001345 76863 rpm-ostree.go:250] Executing rebase from repo path /run/mco-machine-os-content/os-content-521469848/srv/repo with customImageURL pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce and checksum 57356ce06823a842bbb83e2ec652f7003f373d421589a0d37fa842ba918b170d
I0726 20:48:09.001373 76863 update.go:1859] Running: rpm-ostree rebase --experimental /run/mco-machine-os-content/os-content-521469848/srv/repo:57356ce06823a842bbb83e2ec652f7003f373d421589a0d37fa842ba918b170d --custom-origin-url pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865d35cc4252422e66ff7d638530f61d59326f3a519283d9638813fb730ff4ce --custom-origin-description Managed by machine-config-operator
0 metadata, 0 content objects imported; 0 bytes content written
Staging deployment... done
Upgraded:
cri-o 1.21.1-12.rhaos4.8.git30ca719.el8 -> 1.21.2-5.rhaos4.8.gitb27d974.el8
kernel 4.18.0-305.7.1.el8_4 -> 4.18.0-305.10.2.el8_4
kernel-core 4.18.0-305.7.1.el8_4 -> 4.18.0-305.10.2.el8_4
kernel-modules 4.18.0-305.7.1.el8_4 -> 4.18.0-305.10.2.el8_4
kernel-modules-extra 4.18.0-305.7.1.el8_4 -> 4.18.0-305.10.2.el8_4
openshift-hyperkube 4.8.0-202107040818.p0.git.f36aa36.assembly.stream.el8 -> 4.8.0-202107161820.p0.git.051ac4f.assembly.stream.el8
systemd 239-45.el8_4.1 -> 239-45.el8_4.2
systemd-journal-remote 239-45.el8_4.1 -> 239-45.el8_4.2
systemd-libs 239-45.el8_4.1 -> 239-45.el8_4.2
systemd-pam 239-45.el8_4.1 -> 239-45.el8_4.2
systemd-udev 239-45.el8_4.1 -> 239-45.el8_4.2
Run "systemctl reboot" to start a reboot
```
After a reboot the cluster showed "upgraded".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment