[must-gather ] OUT 2025-11-25T00:16:56.597000782Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-25T00:16:56.64913053Z namespace/openshift-must-gather-sddgc created [must-gather ] OUT 2025-11-25T00:16:56.655122807Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-z69tj created [must-gather ] OUT 2025-11-25T00:16:56.835750776Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-46rkf] OUT 2025-11-25T00:18:34.247402653Z waiting for gather to complete [must-gather-46rkf] OUT 2025-11-25T00:18:34.25052739Z downloading gather output [must-gather-46rkf] OUT 2025-11-25T00:18:34.546059994Z receiving incremental file list [must-gather-46rkf] OUT 2025-11-25T00:18:34.552789939Z ./ [must-gather-46rkf] OUT 2025-11-25T00:18:34.553299782Z must-gather.tar.xz [must-gather-46rkf] OUT 2025-11-25T00:18:34.617173408Z packagemanifests [must-gather-46rkf] OUT 2025-11-25T00:18:35.406109149Z run.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.406307664Z version [must-gather-46rkf] OUT 2025-11-25T00:18:35.406426807Z crd/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.406453047Z csv/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.406502208Z ctlplane/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.406632572Z ctlplane/endpoints [must-gather-46rkf] OUT 2025-11-25T00:18:35.406768965Z ctlplane/services [must-gather-46rkf] OUT 2025-11-25T00:18:35.406848117Z ctlplane/ovn/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.406887778Z ctlplane/rabbitmq/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.40697868Z dbs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.407027871Z namespaces/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.407059712Z namespaces/openshift-machine-api/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.407315998Z namespaces/openshift-machine-api/all_resources.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.408424516Z namespaces/openshift-machine-api/events.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.408701742Z namespaces/openshift-machine-api/nad.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.408859816Z namespaces/openshift-machine-api/pvc.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.408927378Z namespaces/openshift-machine-api/buildconfig/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408932678Z namespaces/openshift-machine-api/cert/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408937288Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408941378Z namespaces/openshift-machine-api/configmaps/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408949308Z namespaces/openshift-machine-api/cronjobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408954189Z namespaces/openshift-machine-api/daemonset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.408958339Z namespaces/openshift-machine-api/deployments/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409052161Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.409255186Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.40941034Z namespaces/openshift-machine-api/imagestream/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.40943486Z namespaces/openshift-machine-api/installplans/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409448791Z namespaces/openshift-machine-api/issuer/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409454211Z namespaces/openshift-machine-api/jobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409458451Z namespaces/openshift-machine-api/pods/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409501422Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.409583874Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-46rkf] OUT 2025-11-25T00:18:35.409807589Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.40983232Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.409982864Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.410045095Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-46rkf] OUT 2025-11-25T00:18:35.410196429Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.41025722Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.410421094Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.410589559Z namespaces/openshift-machine-api/replicaset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.41064084Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.410797624Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.410868715Z namespaces/openshift-machine-api/routes/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.410874866Z namespaces/openshift-machine-api/services/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.410945107Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.411235644Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.411336257Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.41145642Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.411728937Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.41185147Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.411969222Z namespaces/openshift-machine-api/statefulset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.411975783Z namespaces/openshift-machine-api/subscriptions/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.411979893Z namespaces/openstack-operators/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412057945Z namespaces/openstack-operators/all_resources.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412139207Z namespaces/openstack-operators/events.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412244229Z namespaces/openstack-operators/nad.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412410743Z namespaces/openstack-operators/pvc.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412459004Z namespaces/openstack-operators/buildconfig/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412464415Z namespaces/openstack-operators/cert/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412468305Z namespaces/openstack-operators/certificaterequests/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412479985Z namespaces/openstack-operators/configmaps/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412484175Z namespaces/openstack-operators/cronjobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412488195Z namespaces/openstack-operators/daemonset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412492035Z namespaces/openstack-operators/deployments/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412496585Z namespaces/openstack-operators/imagestream/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412500375Z namespaces/openstack-operators/installplans/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412504206Z namespaces/openstack-operators/issuer/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412512566Z namespaces/openstack-operators/jobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412516616Z namespaces/openstack-operators/pods/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412540716Z namespaces/openstack-operators/pods/-describe [must-gather-46rkf] OUT 2025-11-25T00:18:35.412621088Z namespaces/openstack-operators/pods/logs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412631139Z namespaces/openstack-operators/replicaset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412635199Z namespaces/openstack-operators/routes/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412639579Z namespaces/openstack-operators/services/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412657809Z namespaces/openstack-operators/statefulset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.412664959Z namespaces/openstack-operators/subscriptions/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.41266931Z namespaces/openstack/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.41270332Z namespaces/openstack/all_resources.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412807673Z namespaces/openstack/events.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.412919796Z namespaces/openstack/nad.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.413045959Z namespaces/openstack/pvc.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.41310318Z namespaces/openstack/buildconfig/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.41310963Z namespaces/openstack/cert/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413114741Z namespaces/openstack/certificaterequests/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413120741Z namespaces/openstack/configmaps/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413126621Z namespaces/openstack/cronjobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413131511Z namespaces/openstack/daemonset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413136041Z namespaces/openstack/deployments/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413143411Z namespaces/openstack/imagestream/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413149161Z namespaces/openstack/installplans/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413160282Z namespaces/openstack/issuer/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413165492Z namespaces/openstack/jobs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413169852Z namespaces/openstack/pods/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413214073Z namespaces/openstack/pods/-describe [must-gather-46rkf] OUT 2025-11-25T00:18:35.413248044Z namespaces/openstack/pods/logs/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413259404Z namespaces/openstack/replicaset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413269684Z namespaces/openstack/routes/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413279995Z namespaces/openstack/services/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413291335Z namespaces/openstack/statefulset/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413303615Z namespaces/openstack/subscriptions/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413334486Z network/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.413357996Z network/l2advertisement.log [must-gather-46rkf] OUT 2025-11-25T00:18:35.414133076Z network/ipaddresspools/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414163386Z network/nnce/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414172836Z network/nncp/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414181957Z network/nns/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414190667Z nodes/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414393282Z nodes/crc.describe [must-gather-46rkf] OUT 2025-11-25T00:18:35.414403802Z nodes/crc.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.414413042Z sos-reports/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414421653Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-46rkf] OUT 2025-11-25T00:18:35.414628488Z sos-reports/_all_nodes/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414644598Z webhooks/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414653748Z webhooks/mutating/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414662929Z webhooks/validating/ [must-gather-46rkf] OUT 2025-11-25T00:18:35.414833893Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.414851303Z webhooks/validating/multus.openshift.io.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.414880314Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-46rkf] OUT 2025-11-25T00:18:35.420941342Z [must-gather-46rkf] OUT 2025-11-25T00:18:35.420974473Z sent 1,129 bytes received 26,360,767 bytes 17,574,597.33 bytes/sec [must-gather-46rkf] OUT 2025-11-25T00:18:35.420984643Z total size is 183,233,356 speedup is 6.95 [must-gather ] OUT 2025-11-25T00:18:35.47062614Z namespace/openshift-must-gather-sddgc deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/route-controller-manager: observed generation is 17, desired generation is 18. clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing