OCP 4 install failure because manifest had version

ocs4-install-error-manifest-version

While trying to install an OCP v4.6 cluster on 3 bare metal nodes I could not figure out why my bootstrap would listen on 6443 but not on 22623 (machine config).
After many reboots and much debugging I ran across a stopped pod on the bootstrap node.  Looking at the pods logs revealed I had added a maninfest with an invalid manifest version (I built it for a 4.7 cluster but then tried to use it in a 4.6 cluster.  

```[root@ocs-bootstrap ~]# crictl ps -a
CONTAINER           IMAGE                                                                                                                    CREATED             STATE               NAME                             ATTEMPT             POD ID
1254caf68e308       c32243984e519c32cedbe26f1c5b2c486b15788711566590901a4f09df524d8c                                                         45 seconds ago      Exited              machine-config-controller        7                   cab653c29c692```

Here is the error message
```# oc logs 1254caf68e308
error: Missing or incomplete configuration info.  Please point to an existing, complete config file:


  1. Via the command-line flag --kubeconfig
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.
[root@ocs-bootstrap ~]# crictl logs 1254caf68e308
I0419 07:09:48.359793       1 bootstrap.go:40] Version: v4.6.0-202103202154.p0-dirty (3c8a2dc5a61b4b7110bc04f429772bb6b677dff2)
I0419 07:09:48.398774       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-dns-02-config.yml" [1] manifest because of unhandled *v1.DNS
I0419 07:09:48.400091       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-infrastructure-02-config.yml" [1] manifest because of unhandled *v1.Infrastructure
I0419 07:09:48.407975       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-ingress-02-config.yml" [1] manifest because of unhandled *v1.Ingress
I0419 07:09:48.408962       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-network-02-config.yml" [1] manifest because of unhandled *v1.Network
I0419 07:09:48.409537       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-proxy-01-config.yaml" [1] manifest because of unhandled *v1.Proxy
I0419 07:09:48.410264       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-scheduler-02-config.yml" [1] manifest because of unhandled *v1.Scheduler
I0419 07:09:48.417654       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cvo-overrides.yaml" [1] manifest because of unhandled *v1.ClusterVersion
F0419 07:09:48.592860       1 bootstrap.go:47] error running MCC[BOOTSTRAP]: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1```

and the offending snippet from the manifest

```
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 3.2.0```

My install (v4.6.8 on 3-node baremetal) has also got stuck at the same stage, but without the last error message..The API comes up ok, but not the machineconfigserver on port 22623 and the bootstrap fails

[root@bootstrap ~]# crictl logs 614e6d4879da9
I0424 19:16:30.712764 1 bootstrap.go:40] Version: v4.6.0-202012050130.p0-dirty (c470febe19e3b004fb99baa6679e7597f50554c5)
I0424 19:16:30.736128 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-dns-02-config.yml" [1] manifest because of unhandled *v1.DNS
I0424 19:16:30.737002 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-infrastructure-02-config.yml" [1] manifest because of unhandled *v1.Infrastructure
I0424 19:16:30.742540 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-ingress-02-config.yml" [1] manifest because of unhandled *v1.Ingress
I0424 19:16:30.743235 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-network-02-config.yml" [1] manifest because of unhandled *v1.Network
I0424 19:16:30.743583 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-proxy-01-config.yaml" [1] manifest because of unhandled *v1.Proxy
I0424 19:16:30.744048 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-scheduler-02-config.yml" [1] manifest because of unhandled *v1.Scheduler
I0424 19:16:30.752176 1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cvo-overrides.yaml" [1] manifest because of unhandled *v1.ClusterVersion
[root@bootstrap ~]#

I'm curious what your install-config.yaml looks like. However, my first guess would be a mismatch between the version of the openshift-install binary and the rhcos image.

[root@ocp-svc openshift]# cat install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:

• hyperthreading: Enabled
  name: worker
  replicas: 1
  controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
  metadata:
  name: ocplab
  networking:
  clusterNetwork:
  • cidr: 10.128.0.0/14
    hostPrefix: 23
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16
    platform:
    none: {}
    fips: false
    pullSecret: '####'
    additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    ...
    ...
    -----END CERTIFICATE-----
    imageContentSources:
• mirrors:
  • 192.168.5.200:5000/ocp-release/openshift468
    source: quay.io/openshift-release-dev/ocp-release
• mirrors:
  • 192.168.5.200:5000/ocp-release/openshift468
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    [root@ocp-svc openshift]#

The baremetal nodes are VMs created on vSphere and I serve the RHCOS image via PXE/tftp using the below files.

rhcos-4.6.8-x86_64-live-kernel-x86_64
rhcos-4.6.8-x86_64-live-initramfs.x86_64.img
rhcos-4.6.8-x86_64-live-rootfs.x86_64.img

and use the below listed client to generate the manifests and ignition files

[root@ocp-svc openshift]# openshift-install version
openshift-install 4.6.8
built from commit f5ba6239853f0904704c04d8b1c04c78172f1141
release image quay.io/openshift-release-dev/ocp-release@sha256:6ddbf56b7f9776c0498f23a54b65a06b3b846c1012200c5609c4bb716b6bdcdf
[root@ocp-svc openshift]#

Is there a fix for this? I have the same problem