[must-gather ] OUT 2025-11-27T00:18:13.947274937Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-27T00:18:14.012053598Z namespace/openshift-must-gather-7qx4g created [must-gather ] OUT 2025-11-27T00:18:14.019468888Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-rqjl4 created [must-gather ] OUT 2025-11-27T00:18:14.195559832Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-gqb2s] OUT 2025-11-27T00:19:50.683565752Z waiting for gather to complete [must-gather-gqb2s] OUT 2025-11-27T00:19:50.685636564Z downloading gather output [must-gather-gqb2s] OUT 2025-11-27T00:19:50.977485367Z receiving incremental file list [must-gather-gqb2s] OUT 2025-11-27T00:19:50.987133135Z ./ [must-gather-gqb2s] OUT 2025-11-27T00:19:50.987344461Z must-gather.tar.xz [must-gather-gqb2s] OUT 2025-11-27T00:19:51.050377804Z packagemanifests [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849166057Z run.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849393883Z version [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849498697Z crd/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849522697Z csv/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849539438Z ctlplane/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.8496112Z ctlplane/endpoints [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849730154Z ctlplane/services [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849803096Z ctlplane/ovn/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849832697Z ctlplane/rabbitmq/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849860217Z dbs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849887248Z namespaces/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.849907489Z namespaces/openshift-machine-api/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850022922Z namespaces/openshift-machine-api/all_resources.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850196667Z namespaces/openshift-machine-api/events.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850421914Z namespaces/openshift-machine-api/nad.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850560528Z namespaces/openshift-machine-api/pvc.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85062002Z namespaces/openshift-machine-api/buildconfig/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850642151Z namespaces/openshift-machine-api/cert/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850666652Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850683362Z namespaces/openshift-machine-api/configmaps/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850716613Z namespaces/openshift-machine-api/cronjobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850733904Z namespaces/openshift-machine-api/daemonset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850752294Z namespaces/openshift-machine-api/deployments/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.850815106Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.8509633Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851105725Z namespaces/openshift-machine-api/imagestream/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851136096Z namespaces/openshift-machine-api/installplans/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851221918Z namespaces/openshift-machine-api/issuer/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85126917Z namespaces/openshift-machine-api/jobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85128297Z namespaces/openshift-machine-api/pods/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851348042Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851398833Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851419364Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851439105Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851586349Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851649911Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851808786Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.851909209Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852085214Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852230048Z namespaces/openshift-machine-api/replicaset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852318861Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852458675Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852572968Z namespaces/openshift-machine-api/routes/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852596309Z namespaces/openshift-machine-api/services/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852666961Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852806115Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.852939209Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853093464Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853330651Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853488326Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853590019Z namespaces/openshift-machine-api/statefulset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85361205Z namespaces/openshift-machine-api/subscriptions/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85362822Z namespaces/openstack-operators/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853704752Z namespaces/openstack-operators/all_resources.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.853825106Z namespaces/openstack-operators/events.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85394559Z namespaces/openstack-operators/nad.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854107044Z namespaces/openstack-operators/pvc.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854204147Z namespaces/openstack-operators/buildconfig/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854224948Z namespaces/openstack-operators/cert/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854243668Z namespaces/openstack-operators/certificaterequests/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854264959Z namespaces/openstack-operators/configmaps/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85428197Z namespaces/openstack-operators/cronjobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85429854Z namespaces/openstack-operators/daemonset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854314861Z namespaces/openstack-operators/deployments/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854335781Z namespaces/openstack-operators/imagestream/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854358052Z namespaces/openstack-operators/installplans/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854387653Z namespaces/openstack-operators/issuer/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854399443Z namespaces/openstack-operators/jobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854411123Z namespaces/openstack-operators/pods/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854473925Z namespaces/openstack-operators/pods/-describe [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854557918Z namespaces/openstack-operators/pods/logs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854578078Z namespaces/openstack-operators/replicaset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854600599Z namespaces/openstack-operators/routes/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.8546179Z namespaces/openstack-operators/services/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85464581Z namespaces/openstack-operators/statefulset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854700792Z namespaces/openstack-operators/subscriptions/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854720093Z namespaces/openstack/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854739673Z namespaces/openstack/all_resources.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854793885Z namespaces/openstack/events.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.854850816Z namespaces/openstack/nad.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85496947Z namespaces/openstack/pvc.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855040222Z namespaces/openstack/buildconfig/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855053963Z namespaces/openstack/cert/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855065933Z namespaces/openstack/certificaterequests/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855087954Z namespaces/openstack/configmaps/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855100334Z namespaces/openstack/cronjobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855112254Z namespaces/openstack/daemonset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855123385Z namespaces/openstack/deployments/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855134605Z namespaces/openstack/imagestream/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855145795Z namespaces/openstack/installplans/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855156826Z namespaces/openstack/issuer/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855167936Z namespaces/openstack/jobs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855179046Z namespaces/openstack/pods/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855190027Z namespaces/openstack/pods/-describe [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855201017Z namespaces/openstack/pods/logs/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855211907Z namespaces/openstack/replicaset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855223188Z namespaces/openstack/routes/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855238938Z namespaces/openstack/services/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855250378Z namespaces/openstack/statefulset/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855262679Z namespaces/openstack/subscriptions/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855274299Z network/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.85529402Z network/l2advertisement.log [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855354832Z network/ipaddresspools/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855368282Z network/nnce/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855380172Z network/nncp/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855391623Z network/nns/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855402953Z nodes/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855433334Z nodes/crc.describe [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855453345Z nodes/crc.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855512706Z sos-reports/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855531837Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855543707Z sos-reports/_all_nodes/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855555298Z webhooks/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855566638Z webhooks/mutating/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855577968Z webhooks/validating/ [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855596469Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.8556533Z webhooks/validating/multus.openshift.io.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.855770764Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-gqb2s] OUT 2025-11-27T00:19:51.861384412Z [must-gather-gqb2s] OUT 2025-11-27T00:19:51.861450304Z sent 1,129 bytes received 26,595,413 bytes 17,731,028.00 bytes/sec [must-gather-gqb2s] OUT 2025-11-27T00:19:51.861461854Z total size is 183,667,591 speedup is 6.91 [must-gather ] OUT 2025-11-27T00:19:51.890201743Z namespace/openshift-must-gather-7qx4g deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/controller-manager: observed generation is 19, desired generation is 20. Progressing: deployment/route-controller-manager: observed generation is 17, desired generation is 18. clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing