[must-gather ] OUT 2026-01-23T20:39:49.928163274Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42606", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0025de318), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/network is progressing: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes) clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2026-01-23T20:39:49.998492969Z namespace/openshift-must-gather-wzmv4 created [must-gather ] OUT 2026-01-23T20:39:50.004022179Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vd5qg created [must-gather ] OUT 2026-01-23T20:39:50.165063052Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-vr842] OUT 2026-01-23T20:41:47.472757548Z waiting for gather to complete [must-gather-vr842] OUT 2026-01-23T20:41:47.476212683Z downloading gather output [must-gather-vr842] OUT 2026-01-23T20:41:47.785494196Z receiving incremental file list [must-gather-vr842] OUT 2026-01-23T20:41:47.792890265Z ./ [must-gather-vr842] OUT 2026-01-23T20:41:47.793611872Z must-gather.tar.xz [must-gather-vr842] OUT 2026-01-23T20:41:47.867137764Z packagemanifests [must-gather-vr842] OUT 2026-01-23T20:41:48.746457739Z run.log [must-gather-vr842] OUT 2026-01-23T20:41:48.746738756Z version [must-gather-vr842] OUT 2026-01-23T20:41:48.746997882Z crd/ [must-gather-vr842] OUT 2026-01-23T20:41:48.747016982Z csv/ [must-gather-vr842] OUT 2026-01-23T20:41:48.747025422Z ctlplane/ [must-gather-vr842] OUT 2026-01-23T20:41:48.747295989Z ctlplane/endpoints [must-gather-vr842] OUT 2026-01-23T20:41:48.747761231Z ctlplane/services [must-gather-vr842] OUT 2026-01-23T20:41:48.747943585Z ctlplane/ovn/ [must-gather-vr842] OUT 2026-01-23T20:41:48.748089739Z ctlplane/rabbitmq/ [must-gather-vr842] OUT 2026-01-23T20:41:48.748203061Z dbs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.748349145Z namespaces/ [must-gather-vr842] OUT 2026-01-23T20:41:48.748481688Z namespaces/openshift-machine-api/ [must-gather-vr842] OUT 2026-01-23T20:41:48.748683873Z namespaces/openshift-machine-api/all_resources.log [must-gather-vr842] OUT 2026-01-23T20:41:48.74893728Z namespaces/openshift-machine-api/events.log [must-gather-vr842] OUT 2026-01-23T20:41:48.749277598Z namespaces/openshift-machine-api/nad.log [must-gather-vr842] OUT 2026-01-23T20:41:48.749530764Z namespaces/openshift-machine-api/pvc.log [must-gather-vr842] OUT 2026-01-23T20:41:48.749735229Z namespaces/openshift-machine-api/buildconfig/ [must-gather-vr842] OUT 2026-01-23T20:41:48.749754969Z namespaces/openshift-machine-api/cert/ [must-gather-vr842] OUT 2026-01-23T20:41:48.749762209Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-vr842] OUT 2026-01-23T20:41:48.749768409Z namespaces/openshift-machine-api/configmaps/ [must-gather-vr842] OUT 2026-01-23T20:41:48.74977715Z namespaces/openshift-machine-api/cronjobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.74978306Z namespaces/openshift-machine-api/daemonset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.74978873Z namespaces/openshift-machine-api/deployments/ [must-gather-vr842] OUT 2026-01-23T20:41:48.750281552Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.750569139Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.750821325Z namespaces/openshift-machine-api/imagestream/ [must-gather-vr842] OUT 2026-01-23T20:41:48.750972289Z namespaces/openshift-machine-api/installplans/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751087182Z namespaces/openshift-machine-api/issuer/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751219825Z namespaces/openshift-machine-api/jobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751371828Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751521672Z namespaces/openshift-machine-api/pods/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751652226Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-vr842] OUT 2026-01-23T20:41:48.751855281Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-vr842] OUT 2026-01-23T20:41:48.752099826Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.752298281Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-vr842] OUT 2026-01-23T20:41:48.752557957Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-vr842] OUT 2026-01-23T20:41:48.752729221Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-vr842] OUT 2026-01-23T20:41:48.753019229Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.753241204Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-vr842] OUT 2026-01-23T20:41:48.753760677Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-vr842] OUT 2026-01-23T20:41:48.753952101Z namespaces/openshift-machine-api/replicaset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.754081414Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.754361332Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.754543126Z namespaces/openshift-machine-api/routes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.754663299Z namespaces/openshift-machine-api/services/ [must-gather-vr842] OUT 2026-01-23T20:41:48.754747751Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.754996927Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.75514162Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.755365666Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.755684323Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.755900949Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.756046683Z namespaces/openshift-machine-api/statefulset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.756076963Z namespaces/openshift-machine-api/subscriptions/ [must-gather-vr842] OUT 2026-01-23T20:41:48.756102914Z namespaces/openshift-operators/ [must-gather-vr842] OUT 2026-01-23T20:41:48.756212227Z namespaces/openshift-operators/all_resources.log [must-gather-vr842] OUT 2026-01-23T20:41:48.756448862Z namespaces/openshift-operators/events.log [must-gather-vr842] OUT 2026-01-23T20:41:48.756699648Z namespaces/openshift-operators/nad.log [must-gather-vr842] OUT 2026-01-23T20:41:48.756961914Z namespaces/openshift-operators/pvc.log [must-gather-vr842] OUT 2026-01-23T20:41:48.757033766Z namespaces/openshift-operators/buildconfig/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757094158Z namespaces/openshift-operators/cert/ [must-gather-vr842] OUT 2026-01-23T20:41:48.75716854Z namespaces/openshift-operators/certificaterequests/ [must-gather-vr842] OUT 2026-01-23T20:41:48.75717659Z namespaces/openshift-operators/configmaps/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757180971Z namespaces/openshift-operators/cronjobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757185041Z namespaces/openshift-operators/daemonset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757189141Z namespaces/openshift-operators/deployments/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757193441Z namespaces/openshift-operators/imagestream/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757197391Z namespaces/openshift-operators/installplans/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757201451Z namespaces/openshift-operators/issuer/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757206231Z namespaces/openshift-operators/jobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757210421Z namespaces/openshift-operators/poddisruptionbudgets/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757214261Z namespaces/openshift-operators/pods/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757221632Z namespaces/openshift-operators/pods/-describe [must-gather-vr842] OUT 2026-01-23T20:41:48.757666972Z namespaces/openshift-operators/pods/logs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757680412Z namespaces/openshift-operators/replicaset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757684452Z namespaces/openshift-operators/routes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757688563Z namespaces/openshift-operators/services/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757692463Z namespaces/openshift-operators/statefulset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757696313Z namespaces/openshift-operators/subscriptions/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757700203Z namespaces/openstack-operators/ [must-gather-vr842] OUT 2026-01-23T20:41:48.757704273Z namespaces/openstack-operators/all_resources.log [must-gather-vr842] OUT 2026-01-23T20:41:48.757708313Z namespaces/openstack-operators/events.log [must-gather-vr842] OUT 2026-01-23T20:41:48.757716093Z namespaces/openstack-operators/nad.log [must-gather-vr842] OUT 2026-01-23T20:41:48.7588002Z namespaces/openstack-operators/pvc.log [must-gather-vr842] OUT 2026-01-23T20:41:48.75881144Z namespaces/openstack-operators/buildconfig/ [must-gather-vr842] OUT 2026-01-23T20:41:48.75881571Z namespaces/openstack-operators/cert/ [must-gather-vr842] OUT 2026-01-23T20:41:48.75881979Z namespaces/openstack-operators/certificaterequests/ [must-gather-vr842] OUT 2026-01-23T20:41:48.75882425Z namespaces/openstack-operators/configmaps/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758834521Z namespaces/openstack-operators/cronjobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758838491Z namespaces/openstack-operators/daemonset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758842791Z namespaces/openstack-operators/deployments/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758847021Z namespaces/openstack-operators/imagestream/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758850911Z namespaces/openstack-operators/installplans/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758854811Z namespaces/openstack-operators/issuer/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758858811Z namespaces/openstack-operators/jobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758862841Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758866751Z namespaces/openstack-operators/pods/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758870681Z namespaces/openstack-operators/pods/-describe [must-gather-vr842] OUT 2026-01-23T20:41:48.758874822Z namespaces/openstack-operators/pods/logs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758879082Z namespaces/openstack-operators/replicaset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758883192Z namespaces/openstack-operators/routes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758887262Z namespaces/openstack-operators/services/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758891212Z namespaces/openstack-operators/statefulset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758895182Z namespaces/openstack-operators/subscriptions/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758900492Z namespaces/openstack/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758904802Z namespaces/openstack/all_resources.log [must-gather-vr842] OUT 2026-01-23T20:41:48.758908722Z namespaces/openstack/events.log [must-gather-vr842] OUT 2026-01-23T20:41:48.758912952Z namespaces/openstack/nad.log [must-gather-vr842] OUT 2026-01-23T20:41:48.758916943Z namespaces/openstack/pvc.log [must-gather-vr842] OUT 2026-01-23T20:41:48.758920793Z namespaces/openstack/buildconfig/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758924653Z namespaces/openstack/cert/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758928553Z namespaces/openstack/certificaterequests/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758932543Z namespaces/openstack/configmaps/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758936403Z namespaces/openstack/cronjobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758940403Z namespaces/openstack/daemonset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758944493Z namespaces/openstack/deployments/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758949373Z namespaces/openstack/imagestream/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758953493Z namespaces/openstack/installplans/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758957544Z namespaces/openstack/issuer/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758962394Z namespaces/openstack/jobs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758966694Z namespaces/openstack/poddisruptionbudgets/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758970544Z namespaces/openstack/pods/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758974474Z namespaces/openstack/pods/-describe [must-gather-vr842] OUT 2026-01-23T20:41:48.758978354Z namespaces/openstack/pods/logs/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758986134Z namespaces/openstack/replicaset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758990394Z namespaces/openstack/routes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758994284Z namespaces/openstack/services/ [must-gather-vr842] OUT 2026-01-23T20:41:48.758998484Z namespaces/openstack/statefulset/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759004675Z namespaces/openstack/subscriptions/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759008685Z network/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759012855Z network/l2advertisement.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.759019545Z network/ipaddresspools/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759024045Z network/nnce/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759028175Z network/nncp/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759035695Z network/nns/ [must-gather-vr842] OUT 2026-01-23T20:41:48.759039815Z nodes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762393467Z nodes/crc.describe [must-gather-vr842] OUT 2026-01-23T20:41:48.762406237Z nodes/crc.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.762410527Z sos-reports/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762414537Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-vr842] OUT 2026-01-23T20:41:48.762418737Z sos-reports/_all_nodes/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762422697Z webhooks/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762426648Z webhooks/mutating/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762430338Z webhooks/validating/ [must-gather-vr842] OUT 2026-01-23T20:41:48.762434048Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.762437908Z webhooks/validating/multus.openshift.io.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.762441828Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-vr842] OUT 2026-01-23T20:41:48.765274677Z [must-gather-vr842] OUT 2026-01-23T20:41:48.765295717Z sent 1,312 bytes received 30,074,334 bytes 20,050,430.67 bytes/sec [must-gather-vr842] OUT 2026-01-23T20:41:48.765302248Z total size is 212,036,791 speedup is 7.05 [must-gather ] OUT 2026-01-23T20:41:48.793454183Z namespace/openshift-must-gather-wzmv4 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing