LAST SEEN TYPE REASON OBJECT MESSAGE 57m Normal Scheduled pod/cluster-autoscaler-operator-7f88444875-zwhqs Successfully assigned openshift-machine-api/cluster-autoscaler-operator-7f88444875-zwhqs to master-0 57m Normal Scheduled pod/machine-api-operator-7486ff55f-tnhrb Successfully assigned openshift-machine-api/machine-api-operator-7486ff55f-tnhrb to master-0 61m Normal Scheduled pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Successfully assigned openshift-machine-api/control-plane-machine-set-operator-66f4cc99d4-6sv72 to master-0 57m Normal Scheduled pod/cluster-baremetal-operator-5fdc576499-xlwrx Successfully assigned openshift-machine-api/cluster-baremetal-operator-5fdc576499-xlwrx to master-0 61m Normal ScalingReplicaSet deployment/control-plane-machine-set-operator Scaled up replica set control-plane-machine-set-operator-66f4cc99d4 to 1 61m Normal SuccessfulCreate replicaset/control-plane-machine-set-operator-66f4cc99d4 Created pod: control-plane-machine-set-operator-66f4cc99d4-6sv72 60m Warning FailedCreatePodSandBox pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_control-plane-machine-set-operator-66f4cc99d4-6sv72_openshift-machine-api_a6a2308d-6d6f-4e8d-a6db-d931a172ed55_0(1429244ef41dd945648a82f1e80bf259ec2c7f0ce876d9eb07c88ec3955beb1c): error adding pod openshift-machine-api_control-plane-machine-set-operator-66f4cc99d4-6sv72 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"1429244ef41dd945648a82f1e80bf259ec2c7f0ce876d9eb07c88ec3955beb1c" Netns:"/var/run/netns/7b05cf23-8fac-4342-bfd7-4d90daaff1d9" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-machine-api;K8S_POD_NAME=control-plane-machine-set-operator-66f4cc99d4-6sv72;K8S_POD_INFRA_CONTAINER_ID=1429244ef41dd945648a82f1e80bf259ec2c7f0ce876d9eb07c88ec3955beb1c;K8S_POD_UID=a6a2308d-6d6f-4e8d-a6db-d931a172ed55" Path:"" ERRORED: error configuring pod [openshift-machine-api/control-plane-machine-set-operator-66f4cc99d4-6sv72] networking: Multus: [openshift-machine-api/control-plane-machine-set-operator-66f4cc99d4-6sv72/a6a2308d-6d6f-4e8d-a6db-d931a172ed55]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod control-plane-machine-set-operator-66f4cc99d4-6sv72 in out of cluster comm: SetNetworkStatus: failed to update the pod control-plane-machine-set-operator-66f4cc99d4-6sv72 in out of cluster comm: status update failed for pod /: Get "https://api-int.sno.openstack.lab:6443/api/v1/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-66f4cc99d4-6sv72?timeout=1m0s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 59m Warning FailedCreatePodSandBox pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_control-plane-machine-set-operator-66f4cc99d4-6sv72_openshift-machine-api_a6a2308d-6d6f-4e8d-a6db-d931a172ed55_0(a09f61f384797066709e5bedbb80f1ce0121afcabab69989b265eb20582b3294): error adding pod openshift-machine-api_control-plane-machine-set-operator-66f4cc99d4-6sv72 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a09f61f384797066709e5bedbb80f1ce0121afcabab69989b265eb20582b3294" Netns:"/var/run/netns/cccbd0b8-9bdc-4d03-9d54-b38746b1365b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-machine-api;K8S_POD_NAME=control-plane-machine-set-operator-66f4cc99d4-6sv72;K8S_POD_INFRA_CONTAINER_ID=a09f61f384797066709e5bedbb80f1ce0121afcabab69989b265eb20582b3294;K8S_POD_UID=a6a2308d-6d6f-4e8d-a6db-d931a172ed55" Path:"" ERRORED: error configuring pod [openshift-machine-api/control-plane-machine-set-operator-66f4cc99d4-6sv72] networking: Multus: [openshift-machine-api/control-plane-machine-set-operator-66f4cc99d4-6sv72/a6a2308d-6d6f-4e8d-a6db-d931a172ed55]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod control-plane-machine-set-operator-66f4cc99d4-6sv72 in out of cluster comm: SetNetworkStatus: failed to update the pod control-plane-machine-set-operator-66f4cc99d4-6sv72 in out of cluster comm: status update failed for pod /: Get "https://api-int.sno.openstack.lab:6443/api/v1/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-66f4cc99d4-6sv72?timeout=1m0s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)... 58m Normal AddedInterface pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Add eth0 [10.128.0.58/23] from ovn-kubernetes 58m Normal Pulling pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23aa409d98c18a25b5dd3c14b4c5a88eba2c793d020f2deb3bafd58a2225c328" 58m Normal Pulled pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23aa409d98c18a25b5dd3c14b4c5a88eba2c793d020f2deb3bafd58a2225c328" in 2.805s (2.805s including waiting). Image size: 465158513 bytes. 58m Normal LeaderElection lease/control-plane-machine-set-leader control-plane-machine-set-operator-66f4cc99d4-6sv72_f37be60d-ab96-4eea-b52d-76951fade1e4 became leader 57m Normal ScalingReplicaSet deployment/cluster-autoscaler-operator Scaled up replica set cluster-autoscaler-operator-7f88444875 to 1 57m Normal SuccessfulCreate replicaset/cluster-autoscaler-operator-7f88444875 Created pod: cluster-autoscaler-operator-7f88444875-zwhqs 57m Normal ScalingReplicaSet deployment/cluster-baremetal-operator Scaled up replica set cluster-baremetal-operator-5fdc576499 to 1 57m Normal SuccessfulCreate replicaset/cluster-baremetal-operator-5fdc576499 Created pod: cluster-baremetal-operator-5fdc576499-xlwrx 57m Normal SuccessfulCreate replicaset/machine-api-operator-7486ff55f Created pod: machine-api-operator-7486ff55f-tnhrb 57m Normal ScalingReplicaSet deployment/machine-api-operator Scaled up replica set machine-api-operator-7486ff55f to 1 57m Normal Pulling pod/machine-api-operator-7486ff55f-tnhrb Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8a38d71a75c4fa803249cc709d60039d14878e218afd88a86083526ee8f78ad" 57m Normal Created pod/cluster-autoscaler-operator-7f88444875-zwhqs Created container: kube-rbac-proxy 57m Normal Pulling pod/cluster-baremetal-operator-5fdc576499-xlwrx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b294511902fd7a80e135b23895a944570932dc0fab1ee22f296523840740332e" 57m Normal Started pod/cluster-autoscaler-operator-7f88444875-zwhqs Started container kube-rbac-proxy 57m Normal Pulling pod/cluster-autoscaler-operator-7f88444875-zwhqs Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d41c3e944e86b73b4ba0d037ff016562211988f3206b9deb6cc7dccca708248" 57m Normal Pulled pod/cluster-autoscaler-operator-7f88444875-zwhqs Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b03d2897e7cc0e8d0c306acb68ca3d9396d502882c14942faadfdb16bc40e17d" already present on machine 57m Normal AddedInterface pod/cluster-autoscaler-operator-7f88444875-zwhqs Add eth0 [10.128.0.63/23] from ovn-kubernetes 57m Normal AddedInterface pod/machine-api-operator-7486ff55f-tnhrb Add eth0 [10.128.0.60/23] from ovn-kubernetes 57m Normal AddedInterface pod/cluster-baremetal-operator-5fdc576499-xlwrx Add eth0 [10.128.0.64/23] from ovn-kubernetes 57m Normal Pulled pod/machine-api-operator-7486ff55f-tnhrb Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b03d2897e7cc0e8d0c306acb68ca3d9396d502882c14942faadfdb16bc40e17d" already present on machine 57m Normal Started pod/machine-api-operator-7486ff55f-tnhrb Started container kube-rbac-proxy 57m Normal Created pod/machine-api-operator-7486ff55f-tnhrb Created container: kube-rbac-proxy 57m Normal Pulled pod/cluster-baremetal-operator-5fdc576499-xlwrx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b294511902fd7a80e135b23895a944570932dc0fab1ee22f296523840740332e" in 11.085s (11.085s including waiting). Image size: 465302163 bytes. 57m Normal Pulled pod/cluster-autoscaler-operator-7f88444875-zwhqs Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d41c3e944e86b73b4ba0d037ff016562211988f3206b9deb6cc7dccca708248" in 11.007s (11.007s including waiting). Image size: 450855746 bytes. 57m Normal Created pod/cluster-baremetal-operator-5fdc576499-xlwrx Created container: baremetal-kube-rbac-proxy 57m Normal LeaderElection lease/cluster-baremetal-operator cluster-baremetal-operator-5fdc576499-xlwrx_08982910-d136-4a1b-8da1-948ffe3a8b06 became leader 57m Normal LeaderElection lease/cluster-autoscaler-operator-leader cluster-autoscaler-operator-7f88444875-zwhqs_1ab9a35d-df9b-4e83-8392-37a171b26d37 became leader 57m Normal Started pod/cluster-baremetal-operator-5fdc576499-xlwrx Started container baremetal-kube-rbac-proxy 57m Normal FeatureGatesInitialized deployment/machine-api-operator FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "ChunkSizeMiB", "CloudDualStackNodeIPs", "DisableKubeletCloudCredentialProviders", "GCPLabelsTags", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "SetEIPForNLBIngressController", "VSphereControlPlaneMachineSet", "VSphereDriverConfiguration", "VSphereMultiVCenters", "VSphereStaticIPs", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AdditionalRoutingCapabilities", "AutomatedEtcdBackup", "BootcNodeManagement", "CSIDriverSharedResource", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ConsolePluginContentSecurityPolicy", "DNSNameResolver", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GatewayAPI", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MetricsCollectionProfiles", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SignatureStores", "SigstoreImageVerification", "StreamingCollectionEncodingToJSON", "StreamingCollectionEncodingToProtobuf", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} 57m Normal Pulled pod/cluster-baremetal-operator-5fdc576499-xlwrx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b03d2897e7cc0e8d0c306acb68ca3d9396d502882c14942faadfdb16bc40e17d" already present on machine 57m Normal Pulled pod/machine-api-operator-7486ff55f-tnhrb Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8a38d71a75c4fa803249cc709d60039d14878e218afd88a86083526ee8f78ad" in 11.222s (11.222s including waiting). Image size: 856674149 bytes. 49m Normal Created pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Created container: control-plane-machine-set-operator 49m Normal Pulled pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23aa409d98c18a25b5dd3c14b4c5a88eba2c793d020f2deb3bafd58a2225c328" already present on machine 49m Normal Started pod/control-plane-machine-set-operator-66f4cc99d4-6sv72 Started container control-plane-machine-set-operator 48m Warning BackOff pod/cluster-baremetal-operator-5fdc576499-xlwrx Back-off restarting failed container cluster-baremetal-operator in pod cluster-baremetal-operator-5fdc576499-xlwrx_openshift-machine-api(17f5a6f7-07dc-45cc-9db4-810f84b678d1) 48m Normal Pulled pod/cluster-baremetal-operator-5fdc576499-xlwrx Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b294511902fd7a80e135b23895a944570932dc0fab1ee22f296523840740332e" already present on machine 48m Normal Started pod/cluster-baremetal-operator-5fdc576499-xlwrx Started container cluster-baremetal-operator 48m Normal Created pod/cluster-baremetal-operator-5fdc576499-xlwrx Created container: cluster-baremetal-operator 47m Normal Pulled pod/cluster-autoscaler-operator-7f88444875-zwhqs Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d41c3e944e86b73b4ba0d037ff016562211988f3206b9deb6cc7dccca708248" already present on machine 47m Normal Pulled pod/machine-api-operator-7486ff55f-tnhrb Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8a38d71a75c4fa803249cc709d60039d14878e218afd88a86083526ee8f78ad" already present on machine 47m Normal Started pod/cluster-autoscaler-operator-7f88444875-zwhqs Started container cluster-autoscaler-operator 47m Normal FeatureGatesInitialized deployment/machine-api-operator FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "ChunkSizeMiB", "CloudDualStackNodeIPs", "DisableKubeletCloudCredentialProviders", "GCPLabelsTags", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NewOLM", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "SetEIPForNLBIngressController", "VSphereControlPlaneMachineSet", "VSphereDriverConfiguration", "VSphereMultiVCenters", "VSphereStaticIPs", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AdditionalRoutingCapabilities", "AutomatedEtcdBackup", "BootcNodeManagement", "CSIDriverSharedResource", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ConsolePluginContentSecurityPolicy", "DNSNameResolver", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GatewayAPI", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MetricsCollectionProfiles", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SignatureStores", "SigstoreImageVerification", "StreamingCollectionEncodingToJSON", "StreamingCollectionEncodingToProtobuf", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} 47m Normal LeaderElection lease/cluster-autoscaler-operator-leader cluster-autoscaler-operator-7f88444875-zwhqs_b474f6b3-37b7-417e-8a08-d875d379ef2a became leader 47m Normal Created pod/cluster-autoscaler-operator-7f88444875-zwhqs Created container: cluster-autoscaler-operator 47m Normal Started pod/machine-api-operator-7486ff55f-tnhrb Started container machine-api-operator 47m Normal Created pod/machine-api-operator-7486ff55f-tnhrb Created container: machine-api-operator 45m Warning FailedMount pod/cluster-autoscaler-operator-7f88444875-zwhqs MountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the condition 45m Warning FailedMount pod/cluster-baremetal-operator-5fdc576499-xlwrx MountVolume.SetUp failed for volume "images" : failed to sync configmap cache: timed out waiting for the condition 45m Warning FailedMount pod/cluster-baremetal-operator-5fdc576499-xlwrx MountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the condition 45m Warning FailedMount pod/cluster-baremetal-operator-5fdc576499-xlwrx MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition 45m Warning FailedMount pod/cluster-autoscaler-operator-7f88444875-zwhqs MountVolume.SetUp failed for volume "auth-proxy-config" : failed to sync configmap cache: timed out waiting for the condition 45m Warning FailedMount pod/cluster-baremetal-operator-5fdc576499-xlwrx MountVolume.SetUp failed for volume "cluster-baremetal-operator-tls" : failed to sync secret cache: timed out waiting for the condition 45m Warning FailedMount pod/machine-api-operator-7486ff55f-tnhrb MountVolume.SetUp failed for volume "images" : failed to sync configmap cache: timed out waiting for the condition 45m Warning FailedMount pod/machine-api-operator-7486ff55f-tnhrb MountVolume.SetUp failed for volume "machine-api-operator-tls" : failed to sync secret cache: timed out waiting for the condition 45m Warning FailedMount pod/machine-api-operator-7486ff55f-tnhrb MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition 44m Normal LeaderElection lease/control-plane-machine-set-leader control-plane-machine-set-operator-66f4cc99d4-6sv72_a4a90346-c8b2-4ff2-9174-b30f3b3a8155 became leader 44m Normal LeaderElection lease/cluster-baremetal-operator cluster-baremetal-operator-5fdc576499-xlwrx_5ee6d9cb-7321-4e88-9da0-401a7123b3ef became leader