Skip to content

Instantly share code, notes, and snippets.

@berinle

berinle/gist:1c2bc8ffadb4b2f74909a2f010f4bffa

Created May 19, 2023 23:13
Show Gist options
  • Save berinle/1c2bc8ffadb4b2f74909a2f010f4bffa to your computer and use it in GitHub Desktop.
Save berinle/1c2bc8ffadb4b2f74909a2f010f4bffa to your computer and use it in GitHub Desktop.
Download ZIP
wcp_deactivate_troubleshoot
Raw
gistfile1.txt
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:310] [opID=64deebef] Processing cluster sync: "99f81a0c-6421-4c01-80b5-bf3111015bd1"
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:314] [opID=64deebef] Attempting to sync supervisor 99f81a0c-6421-4c01-80b5-bf3111015bd1 state with desired state.
2023-05-19T15:34:35.481Z debug wcp [logger/trace.go:77] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] [BEGIN] [kubelifecycle.(*Controller).syncKubeInstanceState:356] supervisor=99f81a0c-6421-4c01-80b5-bf3111015bd1
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:540] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] creating service accounts for cluster domain-c13
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:2161] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Will create VMOperator service account
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:2164] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Will create AppPlatform service account
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:2174] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Will create NSOperator service account
2023-05-19T15:34:35.481Z debug wcp [kubelifecycle/controller.go:2188] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Creating the following service accounts: [0 1 3 4]
2023-05-19T15:34:35.481Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.481Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.481Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.481Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.481Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.492Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-cluster-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60 false 1005 true}] on entity ClusterComputeResource:domain-c13
2023-05-19T15:34:35.496Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-cluster-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60 false 1004 true}] on entity Folder:group-d1
2023-05-19T15:34:35.502Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-cluster-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60 false 1005 true}] on entity ResourcePool:resgroup-241
2023-05-19T15:34:35.507Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-cluster-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60 false 1005 true}] on entity Folder:group-v242
2023-05-19T15:34:35.507Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.516Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1004 true}] on entity Folder:group-d1
2023-05-19T15:34:35.520Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1021 true}] on entity Datacenter:datacenter-8
2023-05-19T15:34:35.523Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1021 true}] on entity ResourcePool:resgroup-241
2023-05-19T15:34:35.527Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1021 true}] on entity Folder:group-v242
2023-05-19T15:34:35.527Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.527Z debug wcp [rest/permission_client.go:38] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Permission client already exists, reuse it.
2023-05-19T15:34:35.575Z debug wcp [auth/permission.go:76] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Set global permission ID=username=berinle.xyz\wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60;doc=urn:acl:global:permissions on principal wcp-vmop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60.
2023-05-19T15:34:35.575Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.586Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-appplatform-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1008 true}] on entity Folder:group-d1
2023-05-19T15:34:35.592Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-appplatform-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1007 true}] on entity ClusterComputeResource:domain-c13
2023-05-19T15:34:35.592Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.592Z debug wcp [rest/permission_client.go:38] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Permission client already exists, reuse it.
2023-05-19T15:34:35.652Z debug wcp [auth/permission.go:76] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Set global permission ID=username=berinle.xyz\wcp-appplatform-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60;doc=urn:acl:global:permissions on principal wcp-appplatform-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60.
2023-05-19T15:34:35.652Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.662Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-nsop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1023 true}] on entity Folder:group-d1
2023-05-19T15:34:35.665Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> wcp-nsop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60@berinle.xyz false 1023 true}] on entity ClusterComputeResource:domain-c13
2023-05-19T15:34:35.665Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.665Z debug wcp [rest/permission_client.go:38] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Permission client already exists, reuse it.
2023-05-19T15:34:35.702Z debug wcp [auth/permission.go:76] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Set global permission ID=username=berinle.xyz\wcp-nsop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60;doc=urn:acl:global:permissions on principal wcp-nsop-user-99f81a0c-6421-4c01-80b5-bf3111015bd1-69fa95ea-cef4-4c98-bc57-2bc46a4b4a60.
2023-05-19T15:34:35.702Z debug wcp [kubelifecycle/identity.go:317] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] VM Image Registry FSS is not set. setImageRegistryOperatorServiceAccountPermissions is a NOOP
2023-05-19T15:34:35.702Z debug wcp [ssolib/vmafd.go:32] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Got cached machine ID: 69fa95ea-cef4-4c98-bc57-2bc46a4b4a60
2023-05-19T15:34:35.710Z debug wcp [vclib/authz.go:50] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Successfully set permissions [{{} <nil> [email protected] false 1027 true}] on entity ClusterComputeResource:domain-c13
-- VISUAL LINE -- 1,1 Top
2023-05-19T15:34:35.796Z debug wcp [logger/trace.go:77] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] [BEGIN] [lifecycle/manager.(*Controller).Do:97] cluster=domain-c13
2023-05-19T15:34:35.796Z debug wcp [kubelifecycle/kube_instance.go:5094] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] refreshing all current agencies
2023-05-19T15:34:35.806Z debug wcp [kubelifecycle/eam_utils.go:299] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Looking for agencies matching 'vmware-vsc-apiserver' in set of 4
2023-05-19T15:34:35.810Z debug wcp [kubelifecycle/eam_utils.go:327] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Inspecting agency 'vCLS'
2023-05-19T15:34:35.814Z debug wcp [kubelifecycle/eam_utils.go:327] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Inspecting agency 'vmware-vsc-apiserver-ms8cfm'
2023-05-19T15:34:35.814Z debug wcp [kubelifecycle/eam_utils.go:354] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Refreshing info for agency Agency:3f1db871-707d-4414-96e9-856ae3a3ce2f in cluster ClusterComputeResource:domain-c13
2023-05-19T15:34:35.842Z debug wcp [kubelifecycle/eam_utils.go:327] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Inspecting agency 'vmware-vsc-apiserver-82mcsd'
2023-05-19T15:34:35.842Z debug wcp [kubelifecycle/eam_utils.go:354] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Refreshing info for agency Agency:710d69de-e36d-4d53-9dd2-4e98c842e3e5 in cluster ClusterComputeResource:domain-c13
2023-05-19T15:34:35.861Z debug wcp [kubelifecycle/eam_utils.go:327] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Inspecting agency 'vmware-vsc-apiserver-cqlxl5'
2023-05-19T15:34:35.861Z debug wcp [kubelifecycle/eam_utils.go:354] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Refreshing info for agency Agency:8edcc256-7b60-4e55-af62-86d16c44033c in cluster ClusterComputeResource:domain-c13
2023-05-19T15:34:35.881Z info wcp [kubelifecycle/eam_utils.go:372] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] finished refreshing current agencies: ([]kubelifecycle.EAMAgencyInfoIf)[(*kubelifecycle.EAMAgencyInfo){agency:(*types.ManagedObjectReference)Agency:3f1db871-707d-4414-96e9-856ae3a3ce2f agent:(*types.ManagedObjectReference)Agent:59367a9e-92ee-48f9-9e64-44fddfa37869 agentInfo:(*types.AgentRuntimeInfo){EamObjectRuntimeInfo:(types.EamObjectRuntimeInfo){DynamicData:(types.DynamicData){} Status:(string)green Issue:([]types.BaseIssue)<nil> GoalState:(string)enabled Entity:(types.ManagedObjectReference)Agent:59367a9e-92ee-48f9-9e64-44fddfa37869} VmPowerState:(types.VirtualMachinePowerState)poweredOn ReceivingHeartBeat:(bool)false Host:(*types.ManagedObjectReference)<nil> Vm:(*types.ManagedObjectReference)VirtualMachine:vm-246 VmIp:(string) VmName:(string) EsxAgentResourcePool:(*types.ManagedObjectReference)<nil> EsxAgentFolder:(*types.ManagedObjectReference)<nil> InstalledBulletin:([]string)<nil> InstalledVibs:([]types.VibVibInfo)<nil> Agency:(*types.ManagedObjectReference)Agency:3f1db871-707d-4414-96e9-856ae3a3ce2f VmHook:(*types.AgentVmHook)<nil>} cluster:(types.ManagedObjectReference)ClusterComputeResource:domain-c13 zone:(instanceconfig.FaultDomainZone){ID:(instanceconfig.FaultDomainZoneID)domain-c13 ClusterComputeResources:([]types.ManagedObjectReference)[ClusterComputeResource:domain-c13]}} (*kubelifecycle.EAMAgencyInfo){agency:(*types.ManagedObjectReference)Agency:710d69de-e36d-4d53-9dd2-4e98c842e3e5 agent:(*types.ManagedObjectReference)Agent:51c1a93c-bdaa-4cf4-ab23-4d8175a93f9c agentInfo:(*types.AgentRuntimeInfo){EamObjectRuntimeInfo:(types.EamObjectRuntimeInfo){DynamicData:(types.DynamicData){} Status:(string)green Issue:([]types.BaseIssue)<nil> GoalState:(string)enabled Entity:(types.ManagedObjectReference)Agent:51c1a93c-bdaa-4cf4-ab23-4d8175a93f9c} VmPowerState:(types.VirtualMachinePowerState)poweredOn ReceivingHeartBeat:(bool)false Host:(*types.ManagedObjectReference)<nil> Vm:(*types.ManagedObjectReference)VirtualMachine:vm-248 VmIp:(string) VmName:(string) EsxAgentResourcePool:(*types.ManagedObjectReference)<nil> EsxAgentFolder:(*types.ManagedObjectReference)<nil> InstalledBulletin:([]string)<nil> InstalledVibs:([]types.VibVibInfo)<nil> Agency:(*types.ManagedObjectReference)Agency:710d69de-e36d-4d53-9dd2-4e98c842e3e5 VmHook:(*types.AgentVmHook)<nil>} cluster:(types.ManagedObjectReference)ClusterComputeResource:domain-c13 zone:(instanceconfig.FaultDomainZone){ID:(instanceconfig.FaultDomainZoneID)domain-c13 ClusterComputeResources:([]types.ManagedObjectReference)[ClusterComputeResource:domain-c13]}} (*kubelifecycle.EAMAgencyInfo){agency:(*types.ManagedObjectReference)Agency:8edcc256-7b60-4e55-af62-86d16c44033c agent:(*types.ManagedObjectReference)Agent:cbaf089d-8113-431e-ad02-38f917842a8b agentInfo:(*types.AgentRuntimeInfo){EamObjectRuntimeInfo:(types.EamObjectRuntimeInfo){DynamicData:(types.DynamicData){} Status:(string)green Issue:([]types.BaseIssue)<nil> GoalState:(string)enabled Entity:(types.ManagedObjectReference)Agent:cbaf089d-8113-431e-ad02-38f917842a8b} VmPowerState:(types.VirtualMachinePowerState)poweredOn ReceivingHeartBeat:(bool)false Host:(*types.ManagedObjectReference)<nil> Vm:(*types.ManagedObjectReference)VirtualMachine:vm-247 VmIp:(string) VmName:(string) EsxAgentResourcePool:(*types.ManagedObjectReference)<nil> EsxAgentFolder:(*types.ManagedObjectReference)<nil> InstalledBulletin:([]string)<nil> InstalledVibs:([]types.VibVibInfo)<nil> Agency:(*types.ManagedObjectReference)Agency:8edcc256-7b60-4e55-af62-86d16c44033c VmHook:(*types.AgentVmHook)<nil>} cluster:(types.ManagedObjectReference)ClusterComputeResource:domain-c13 zone:(instanceconfig.FaultDomainZone){ID:(instanceconfig.FaultDomainZoneID)domain-c13 ClusterComputeResources:([]types.ManagedObjectReference)[ClusterComputeResource:domain-c13]}}]
2023-05-19T15:34:35.890Z debug wcp [rest/client.go:84] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Rest client for vmodl2 API calls exists, checking session validity
2023-05-19T15:34:35.893Z debug wcp [rest/client.go:87] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] Rest client for vmodl2 API calls is still valid.
2023-05-19T15:34:35.893Z debug wcp [kubelifecycle/eam_utils.go:141] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] getting k8s version from master vm VirtualMachine:vm-246
2023-05-19T15:36:35.972Z debug wcp [kubelifecycle/eam_utils.go:159] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] unable to get k8s version from agency &{0xc00e4b8ba0 0xc00e4b9020 0xc0090f9c20 {ClusterComputeResource domain-c13} {domain-c13 [{ClusterComputeResource domain-c13}]}}: Get "https://10.0.10.22:6443/version?timeout=2m0s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-05-19T15:36:35.973Z debug wcp [kubelifecycle/eam_utils.go:141] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] getting k8s version from master vm VirtualMachine:vm-248
2023-05-19T15:38:56.215Z debug wcp [kubelifecycle/eam_utils.go:159] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] unable to get k8s version from agency &{0xc00e4b9a20 0xc00e53a7c0 0xc0073ee690 {ClusterComputeResource domain-c13} {domain-c13 [{ClusterComputeResource domain-c13}]}}: Get "https://10.0.10.23:6443/version?timeout=2m0s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2023-05-19T15:38:56.215Z debug wcp [kubelifecycle/eam_utils.go:141] [opID=64deebef-99f81a0c-6421-4c01-80b5-bf3111015bd1] getting k8s version from master vm VirtualMachine:vm-247
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment