[must-gather ] OUT 2025-12-12T00:19:11.780246405Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.6 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43369", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.December, 12, 0, 15, 22, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00403e9d8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because CustomRouteControllerDegraded: the server is currently unable to handle the request (get routes.route.openshift.io oauth-openshift) WellKnownReadyControllerDegraded: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43369", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.December, 12, 0, 15, 22, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00403e9d8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly) clusteroperator/console is not upgradeable because ConsoleCustomRouteSyncUpgradeable: the server is currently unable to handle the request (delete routes.route.openshift.io console-custom) DownloadsCustomRouteSyncUpgradeable: the server is currently unable to handle the request (delete routes.route.openshift.io downloads-custom) clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/network is progressing: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes) clusteroperator/openshift-apiserver is not available (APIServerDeploymentAvailable: no apiserver.openshift-apiserver pods available on any node. APIServicesAvailable: PreconditionNotReady) because APIServerDeploymentDegraded: 1 of 1 requested instances are unavailable for apiserver.openshift-apiserver (container is not ready in apiserver-7fc54b8dd7-d2bhp pod) clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-12T00:19:11.836746856Z namespace/openshift-must-gather-7z2xz created [must-gather ] OUT 2025-12-12T00:19:11.845177985Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-9ww7n created [must-gather ] OUT 2025-12-12T00:19:11.868020671Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-5jmgx] OUT 2025-12-12T00:20:17.898805502Z waiting for gather to complete [must-gather-5jmgx] OUT 2025-12-12T00:20:17.903035377Z downloading gather output [must-gather-5jmgx] OUT 2025-12-12T00:20:18.144551319Z receiving incremental file list [must-gather-5jmgx] OUT 2025-12-12T00:20:18.150038765Z ./ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.150267281Z must-gather.tar.xz [must-gather-5jmgx] OUT 2025-12-12T00:20:18.224295806Z packagemanifests [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962567347Z run.log [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962696261Z version [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962750912Z crd/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962757932Z ctlplane/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962766122Z ctlplane/endpoints [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962778753Z ctlplane/services [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962786313Z ctlplane/ovn/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962792083Z ctlplane/rabbitmq/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962798023Z dbs/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962805013Z network/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962811363Z network/l2advertisement.log [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962822634Z network/ipaddresspools/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962828304Z network/nnce/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962833814Z network/nncp/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962839814Z network/nns/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962845704Z nodes/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.962928716Z nodes/crc.describe [must-gather-5jmgx] OUT 2025-12-12T00:20:18.96345405Z nodes/crc.yaml [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963492171Z sos-reports/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963546322Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963570953Z sos-reports/_all_nodes/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963582983Z webhooks/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963593853Z webhooks/mutating/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963604303Z webhooks/validating/ [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963614464Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963649725Z webhooks/validating/multus.openshift.io.yaml [must-gather-5jmgx] OUT 2025-12-12T00:20:18.963664905Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-5jmgx] OUT 2025-12-12T00:20:18.966409473Z [must-gather-5jmgx] OUT 2025-12-12T00:20:18.966455734Z sent 330 bytes received 26,998,328 bytes 17,999,105.33 bytes/sec [must-gather-5jmgx] OUT 2025-12-12T00:20:18.966464494Z total size is 186,010,345 speedup is 6.89 [must-gather ] OUT 2025-12-12T00:20:18.998013646Z namespace/openshift-must-gather-7z2xz deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.6 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing