[must-gather ] OUT 2025-12-08T00:20:03.869607437Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.5 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43111", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002790690), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-08T00:20:03.926121803Z namespace/openshift-must-gather-clrck created [must-gather ] OUT 2025-12-08T00:20:03.930541341Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-7j6xt created [must-gather ] OUT 2025-12-08T00:20:03.944349372Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-hrnwr] OUT 2025-12-08T00:21:50.438238374Z waiting for gather to complete [must-gather-hrnwr] OUT 2025-12-08T00:21:50.442846798Z downloading gather output [must-gather-hrnwr] OUT 2025-12-08T00:21:50.678316356Z receiving incremental file list [must-gather-hrnwr] OUT 2025-12-08T00:21:50.69043342Z ./ [must-gather-hrnwr] OUT 2025-12-08T00:21:50.690474562Z must-gather.tar.xz [must-gather-hrnwr] OUT 2025-12-08T00:21:50.745887418Z packagemanifests [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490772155Z run.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490849087Z version [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490888648Z crd/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490901348Z csv/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490913449Z ctlplane/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490941969Z ctlplane/endpoints [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49096483Z ctlplane/services [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49097646Z ctlplane/ovn/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49098788Z ctlplane/rabbitmq/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.490999141Z dbs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491008431Z namespaces/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491017821Z namespaces/openshift-machine-api/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491032662Z namespaces/openshift-machine-api/all_resources.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491114264Z namespaces/openshift-machine-api/events.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491292278Z namespaces/openshift-machine-api/nad.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491652729Z namespaces/openshift-machine-api/pvc.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491668719Z namespaces/openshift-machine-api/buildconfig/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491673469Z namespaces/openshift-machine-api/cert/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491677699Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49168193Z namespaces/openshift-machine-api/configmaps/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49168582Z namespaces/openshift-machine-api/cronjobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49169356Z namespaces/openshift-machine-api/daemonset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.4916974Z namespaces/openshift-machine-api/deployments/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49170129Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49170569Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49170993Z namespaces/openshift-machine-api/imagestream/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49171377Z namespaces/openshift-machine-api/installplans/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491717581Z namespaces/openshift-machine-api/issuer/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491721561Z namespaces/openshift-machine-api/jobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491725421Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491729191Z namespaces/openshift-machine-api/pods/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491733021Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491738571Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-hrnwr] OUT 2025-12-08T00:21:51.491745781Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492770779Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492787679Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49279219Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49279668Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49280094Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492805Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49281996Z namespaces/openshift-machine-api/replicaset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49282407Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492827891Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492831641Z namespaces/openshift-machine-api/routes/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492835481Z namespaces/openshift-machine-api/services/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492839301Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492843061Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492846741Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492850911Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492865502Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492930383Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492974004Z namespaces/openshift-machine-api/statefulset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492978604Z namespaces/openshift-machine-api/subscriptions/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.492983515Z namespaces/openstack-operators/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493044106Z namespaces/openstack-operators/all_resources.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493096398Z namespaces/openstack-operators/events.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.4931885Z namespaces/openstack-operators/nad.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493259552Z namespaces/openstack-operators/pvc.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493303603Z namespaces/openstack-operators/buildconfig/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493310163Z namespaces/openstack-operators/cert/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493315173Z namespaces/openstack-operators/certificaterequests/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493320013Z namespaces/openstack-operators/configmaps/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493325033Z namespaces/openstack-operators/cronjobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493330014Z namespaces/openstack-operators/daemonset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493335004Z namespaces/openstack-operators/deployments/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493344134Z namespaces/openstack-operators/imagestream/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493348704Z namespaces/openstack-operators/installplans/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493352364Z namespaces/openstack-operators/issuer/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493356184Z namespaces/openstack-operators/jobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493359824Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493363715Z namespaces/openstack-operators/pods/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493367845Z namespaces/openstack-operators/pods/-describe [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493373675Z namespaces/openstack-operators/pods/logs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493377475Z namespaces/openstack-operators/replicaset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493381415Z namespaces/openstack-operators/routes/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493387165Z namespaces/openstack-operators/services/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493391235Z namespaces/openstack-operators/statefulset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493394975Z namespaces/openstack-operators/subscriptions/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493398865Z namespaces/openstack/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493429966Z namespaces/openstack/all_resources.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493494518Z namespaces/openstack/events.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49356469Z namespaces/openstack/nad.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493699973Z namespaces/openstack/pvc.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493712664Z namespaces/openstack/buildconfig/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493717104Z namespaces/openstack/cert/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493721684Z namespaces/openstack/certificaterequests/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493725914Z namespaces/openstack/configmaps/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493730464Z namespaces/openstack/cronjobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493735444Z namespaces/openstack/daemonset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493739674Z namespaces/openstack/deployments/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493743924Z namespaces/openstack/imagestream/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493748055Z namespaces/openstack/installplans/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493761005Z namespaces/openstack/issuer/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493765355Z namespaces/openstack/jobs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493770405Z namespaces/openstack/poddisruptionbudgets/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493775015Z namespaces/openstack/pods/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493793606Z namespaces/openstack/pods/-describe [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493870048Z namespaces/openstack/pods/logs/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493903859Z namespaces/openstack/replicaset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493913789Z namespaces/openstack/routes/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493923279Z namespaces/openstack/services/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493932649Z namespaces/openstack/statefulset/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49395957Z namespaces/openstack/subscriptions/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.49397042Z network/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.493982931Z network/l2advertisement.log [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494005711Z network/ipaddresspools/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494015712Z network/nnce/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494036672Z network/nncp/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494046032Z network/nns/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494055323Z nodes/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494064633Z nodes/crc.describe [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494091073Z nodes/crc.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494179957Z sos-reports/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494192917Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494198147Z sos-reports/_all_nodes/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494202777Z webhooks/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494207258Z webhooks/mutating/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494212048Z webhooks/validating/ [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494229548Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494330391Z webhooks/validating/multus.openshift.io.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.494429783Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-hrnwr] OUT 2025-12-08T00:21:51.497526306Z [must-gather-hrnwr] OUT 2025-12-08T00:21:51.497543497Z sent 1,141 bytes received 26,419,046 bytes 17,613,458.00 bytes/sec [must-gather-hrnwr] OUT 2025-12-08T00:21:51.497549017Z total size is 185,567,225 speedup is 7.02 [must-gather ] OUT 2025-12-08T00:21:51.527767128Z namespace/openshift-must-gather-clrck deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.5 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing