OCP 4 install failure because manifest had version

ocs4-install-error-manifest-version

While trying to install an OCP v4.6 cluster on 3 bare metal nodes I could not figure out why my bootstrap would listen on 6443 but not on 22623 (machine config).
After many reboots and much debugging I ran across a stopped pod on the bootstrap node.  Looking at the pods logs revealed I had added a maninfest with an invalid manifest version (I built it for a 4.7 cluster but then tried to use it in a 4.6 cluster.  

```[root@ocs-bootstrap ~]# crictl ps -a
CONTAINER           IMAGE                                                                                                                    CREATED             STATE               NAME                             ATTEMPT             POD ID
1254caf68e308       c32243984e519c32cedbe26f1c5b2c486b15788711566590901a4f09df524d8c                                                         45 seconds ago      Exited              machine-config-controller        7                   cab653c29c692```

Here is the error message
```# oc logs 1254caf68e308
error: Missing or incomplete configuration info.  Please point to an existing, complete config file:


  1. Via the command-line flag --kubeconfig
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.
[root@ocs-bootstrap ~]# crictl logs 1254caf68e308
I0419 07:09:48.359793       1 bootstrap.go:40] Version: v4.6.0-202103202154.p0-dirty (3c8a2dc5a61b4b7110bc04f429772bb6b677dff2)
I0419 07:09:48.398774       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-dns-02-config.yml" [1] manifest because of unhandled *v1.DNS
I0419 07:09:48.400091       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-infrastructure-02-config.yml" [1] manifest because of unhandled *v1.Infrastructure
I0419 07:09:48.407975       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-ingress-02-config.yml" [1] manifest because of unhandled *v1.Ingress
I0419 07:09:48.408962       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-network-02-config.yml" [1] manifest because of unhandled *v1.Network
I0419 07:09:48.409537       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-proxy-01-config.yaml" [1] manifest because of unhandled *v1.Proxy
I0419 07:09:48.410264       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cluster-scheduler-02-config.yml" [1] manifest because of unhandled *v1.Scheduler
I0419 07:09:48.417654       1 bootstrap.go:116] skipping "/etc/mcc/bootstrap/cvo-overrides.yaml" [1] manifest because of unhandled *v1.ClusterVersion
F0419 07:09:48.592860       1 bootstrap.go:47] error running MCC[BOOTSTRAP]: parsing Ignition config failed: unknown version. Supported spec versions: 2.2, 3.0, 3.1```

and the offending snippet from the manifest

```
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 3.2.0```

I'm curious what your install-config.yaml looks like. However, my first guess would be a mismatch between the version of the openshift-install binary and the rhcos image.

[root@ocp-svc openshift]# cat install-config.yaml
apiVersion: v1
baseDomain: example.com
compute:

• hyperthreading: Enabled
  name: worker
  replicas: 1
  controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
  metadata:
  name: ocplab
  networking:
  clusterNetwork:
  • cidr: 10.128.0.0/14
    hostPrefix: 23
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16
    platform:
    none: {}
    fips: false
    pullSecret: '####'
    additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    ...
    ...
    -----END CERTIFICATE-----
    imageContentSources:
• mirrors:
  • 192.168.5.200:5000/ocp-release/openshift468
    source: quay.io/openshift-release-dev/ocp-release
• mirrors:
  • 192.168.5.200:5000/ocp-release/openshift468
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    [root@ocp-svc openshift]#

The baremetal nodes are VMs created on vSphere and I serve the RHCOS image via PXE/tftp using the below files.

rhcos-4.6.8-x86_64-live-kernel-x86_64
rhcos-4.6.8-x86_64-live-initramfs.x86_64.img
rhcos-4.6.8-x86_64-live-rootfs.x86_64.img

and use the below listed client to generate the manifests and ignition files

[root@ocp-svc openshift]# openshift-install version
openshift-install 4.6.8
built from commit f5ba6239853f0904704c04d8b1c04c78172f1141
release image quay.io/openshift-release-dev/ocp-release@sha256:6ddbf56b7f9776c0498f23a54b65a06b3b846c1012200c5609c4bb716b6bdcdf
[root@ocp-svc openshift]#

Is there a fix for this? I have the same problem