[must-gather ] OUT 2025-11-24T17:39:10.328704714Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42860", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002264dc8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/controller-manager: observed generation is 16, desired generation is 17. Progressing: deployment/route-controller-manager: observed generation is 14, desired generation is 15. clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-24T17:39:10.388374592Z namespace/openshift-must-gather-sln46 created [must-gather ] OUT 2025-11-24T17:39:10.392799349Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-7fwvc created [must-gather ] OUT 2025-11-24T17:39:10.407012702Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-dblkb] OUT 2025-11-24T17:40:43.946514847Z waiting for gather to complete [must-gather-dblkb] OUT 2025-11-24T17:40:43.950668587Z downloading gather output [must-gather-dblkb] OUT 2025-11-24T17:40:44.205376401Z receiving incremental file list [must-gather-dblkb] OUT 2025-11-24T17:40:44.210125095Z ./ [must-gather-dblkb] OUT 2025-11-24T17:40:44.21033555Z must-gather.tar.xz [must-gather-dblkb] OUT 2025-11-24T17:40:44.260316166Z packagemanifests [must-gather-dblkb] OUT 2025-11-24T17:40:45.018147685Z run.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.018219926Z version [must-gather-dblkb] OUT 2025-11-24T17:40:45.018278848Z crd/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018292918Z csv/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018297878Z ctlplane/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018313958Z ctlplane/endpoints [must-gather-dblkb] OUT 2025-11-24T17:40:45.01835981Z ctlplane/services [must-gather-dblkb] OUT 2025-11-24T17:40:45.01837418Z ctlplane/ovn/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.01837963Z ctlplane/rabbitmq/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.01838754Z dbs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.01839224Z namespaces/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018405921Z namespaces/openshift-machine-api/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018462242Z namespaces/openshift-machine-api/all_resources.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.018563984Z namespaces/openshift-machine-api/events.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.01876904Z namespaces/openshift-machine-api/nad.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.018822911Z namespaces/openshift-machine-api/pvc.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.018856402Z namespaces/openshift-machine-api/buildconfig/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018860692Z namespaces/openshift-machine-api/cert/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018864842Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018868482Z namespaces/openshift-machine-api/configmaps/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018872213Z namespaces/openshift-machine-api/cronjobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018878153Z namespaces/openshift-machine-api/daemonset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018882193Z namespaces/openshift-machine-api/deployments/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.018906923Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.019002796Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.019077107Z namespaces/openshift-machine-api/imagestream/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019081447Z namespaces/openshift-machine-api/installplans/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019085287Z namespaces/openshift-machine-api/issuer/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019089128Z namespaces/openshift-machine-api/jobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019117538Z namespaces/openshift-machine-api/pods/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019126628Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019134319Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-dblkb] OUT 2025-11-24T17:40:45.019229901Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019244001Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.019352494Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019363234Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-dblkb] OUT 2025-11-24T17:40:45.019461606Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019502117Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.01961841Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.019698642Z namespaces/openshift-machine-api/replicaset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019746623Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.019833735Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.019896086Z namespaces/openshift-machine-api/routes/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019900797Z namespaces/openshift-machine-api/services/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.019920287Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.02003263Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.020087891Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.020174914Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.020329948Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.02041187Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.020470071Z namespaces/openshift-machine-api/statefulset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020478831Z namespaces/openshift-machine-api/subscriptions/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020482841Z namespaces/openstack-operators/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020509422Z namespaces/openstack-operators/all_resources.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.020615714Z namespaces/openstack-operators/events.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.020676086Z namespaces/openstack-operators/nad.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.020793249Z namespaces/openstack-operators/pvc.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.020810189Z namespaces/openstack-operators/buildconfig/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020815459Z namespaces/openstack-operators/cert/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020819189Z namespaces/openstack-operators/certificaterequests/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020825499Z namespaces/openstack-operators/configmaps/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020829279Z namespaces/openstack-operators/cronjobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020833019Z namespaces/openstack-operators/daemonset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02083667Z namespaces/openstack-operators/deployments/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02084034Z namespaces/openstack-operators/imagestream/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02084404Z namespaces/openstack-operators/installplans/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02084798Z namespaces/openstack-operators/issuer/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02085179Z namespaces/openstack-operators/jobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02085553Z namespaces/openstack-operators/pods/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02086161Z namespaces/openstack-operators/pods/-describe [must-gather-dblkb] OUT 2025-11-24T17:40:45.020909701Z namespaces/openstack-operators/pods/logs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020915321Z namespaces/openstack-operators/replicaset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020919481Z namespaces/openstack-operators/routes/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020923092Z namespaces/openstack-operators/services/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020926672Z namespaces/openstack-operators/statefulset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020947922Z namespaces/openstack-operators/subscriptions/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020952632Z namespaces/openstack/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.020964503Z namespaces/openstack/all_resources.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.021030654Z namespaces/openstack/events.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.021080815Z namespaces/openstack/nad.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.021207898Z namespaces/openstack/pvc.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.021221979Z namespaces/openstack/buildconfig/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021226019Z namespaces/openstack/cert/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021229709Z namespaces/openstack/certificaterequests/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021233349Z namespaces/openstack/configmaps/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021254639Z namespaces/openstack/cronjobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021262999Z namespaces/openstack/daemonset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02128058Z namespaces/openstack/deployments/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.02128651Z namespaces/openstack/imagestream/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021325681Z namespaces/openstack/installplans/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021331011Z namespaces/openstack/issuer/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021335451Z namespaces/openstack/jobs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021339821Z namespaces/openstack/pods/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021351222Z namespaces/openstack/pods/-describe [must-gather-dblkb] OUT 2025-11-24T17:40:45.021356982Z namespaces/openstack/pods/logs/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021384462Z namespaces/openstack/replicaset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021394193Z namespaces/openstack/routes/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021404183Z namespaces/openstack/services/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021408833Z namespaces/openstack/statefulset/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021413383Z namespaces/openstack/subscriptions/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021418023Z network/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021426863Z network/l2advertisement.log [must-gather-dblkb] OUT 2025-11-24T17:40:45.021435174Z network/ipaddresspools/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021439434Z network/nnce/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021443854Z network/nncp/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021453444Z network/nns/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021457214Z nodes/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021461084Z nodes/crc.describe [must-gather-dblkb] OUT 2025-11-24T17:40:45.02166665Z nodes/crc.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.021768812Z sos-reports/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021787063Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-dblkb] OUT 2025-11-24T17:40:45.021791353Z sos-reports/_all_nodes/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021795153Z webhooks/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021802883Z webhooks/mutating/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021807483Z webhooks/validating/ [must-gather-dblkb] OUT 2025-11-24T17:40:45.021814313Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.021920886Z webhooks/validating/multus.openshift.io.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.021975687Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-dblkb] OUT 2025-11-24T17:40:45.024746544Z [must-gather-dblkb] OUT 2025-11-24T17:40:45.024794085Z sent 1,129 bytes received 27,259,588 bytes 10,904,286.80 bytes/sec [must-gather-dblkb] OUT 2025-11-24T17:40:45.024799575Z total size is 184,185,392 speedup is 6.76 [must-gather ] OUT 2025-11-24T17:40:45.092263432Z namespace/openshift-must-gather-sln46 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing