[must-gather ] OUT 2025-11-26T11:07:43.039753412Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/network is progressing: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes) clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-26T11:07:43.102331849Z namespace/openshift-must-gather-rbzfh created [must-gather ] OUT 2025-11-26T11:07:43.105434233Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-pl692 created [must-gather ] OUT 2025-11-26T11:07:43.2415062Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-7q2jz] OUT 2025-11-26T11:09:30.596322108Z waiting for gather to complete [must-gather-7q2jz] OUT 2025-11-26T11:09:30.603363828Z downloading gather output [must-gather-7q2jz] OUT 2025-11-26T11:09:30.938922864Z receiving incremental file list [must-gather-7q2jz] OUT 2025-11-26T11:09:30.952972965Z ./ [must-gather-7q2jz] OUT 2025-11-26T11:09:30.953439177Z must-gather.tar.xz [must-gather-7q2jz] OUT 2025-11-26T11:09:31.034571816Z packagemanifests [must-gather-7q2jz] OUT 2025-11-26T11:09:31.775614994Z run.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.775673385Z version [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776001214Z crd/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776017394Z csv/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776021515Z ctlplane/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776367494Z ctlplane/endpoints [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776550388Z ctlplane/services [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776822596Z ctlplane/ovn/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776841346Z ctlplane/rabbitmq/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776846346Z dbs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776851326Z namespaces/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.776855897Z namespaces/cert-manager/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.777212487Z namespaces/cert-manager/all_resources.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.777497254Z namespaces/cert-manager/events.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.777762211Z namespaces/cert-manager/nad.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.777993788Z namespaces/cert-manager/pvc.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778157532Z namespaces/cert-manager/buildconfig/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778168602Z namespaces/cert-manager/cert/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778175072Z namespaces/cert-manager/certificaterequests/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778179652Z namespaces/cert-manager/configmaps/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778184003Z namespaces/cert-manager/cronjobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778188653Z namespaces/cert-manager/daemonset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778196073Z namespaces/cert-manager/deployments/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778200643Z namespaces/cert-manager/imagestream/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778205613Z namespaces/cert-manager/installplans/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778740958Z namespaces/cert-manager/issuer/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778782069Z namespaces/cert-manager/jobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.778788289Z namespaces/cert-manager/pods/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779117508Z namespaces/cert-manager/pods/-describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779338934Z namespaces/cert-manager/pods/logs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779350464Z namespaces/cert-manager/replicaset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779355785Z namespaces/cert-manager/routes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779361405Z namespaces/cert-manager/services/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779366055Z namespaces/cert-manager/statefulset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779371175Z namespaces/cert-manager/subscriptions/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.779375915Z namespaces/openshift-machine-api/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.7799181Z namespaces/openshift-machine-api/all_resources.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.780212438Z namespaces/openshift-machine-api/events.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78068104Z namespaces/openshift-machine-api/nad.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.780917447Z namespaces/openshift-machine-api/pvc.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781107462Z namespaces/openshift-machine-api/buildconfig/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781132623Z namespaces/openshift-machine-api/cert/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781169534Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781174435Z namespaces/openshift-machine-api/configmaps/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781179095Z namespaces/openshift-machine-api/cronjobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781183565Z namespaces/openshift-machine-api/daemonset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781188245Z namespaces/openshift-machine-api/deployments/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.781880393Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78214501Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782355326Z namespaces/openshift-machine-api/imagestream/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782368586Z namespaces/openshift-machine-api/installplans/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782373736Z namespaces/openshift-machine-api/issuer/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782382906Z namespaces/openshift-machine-api/jobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782387607Z namespaces/openshift-machine-api/pods/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782392557Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.782967503Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.783298312Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.783466436Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.783761854Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.783911508Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.784141805Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.784297609Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.784623918Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.784898135Z namespaces/openshift-machine-api/replicaset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78510731Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.785355908Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.785505452Z namespaces/openshift-machine-api/routes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.785541883Z namespaces/openshift-machine-api/services/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.785652466Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.785886212Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.786077007Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.786277832Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.786622741Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.786837508Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.786967671Z namespaces/openshift-machine-api/statefulset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787004012Z namespaces/openshift-machine-api/subscriptions/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787053184Z namespaces/openstack-operators/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787215848Z namespaces/openstack-operators/all_resources.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787428714Z namespaces/openstack-operators/events.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787579958Z namespaces/openstack-operators/nad.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787794673Z namespaces/openstack-operators/pvc.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787889636Z namespaces/openstack-operators/buildconfig/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787940487Z namespaces/openstack-operators/cert/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.787996599Z namespaces/openstack-operators/certificaterequests/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78805138Z namespaces/openstack-operators/configmaps/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788080131Z namespaces/openstack-operators/cronjobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788125062Z namespaces/openstack-operators/daemonset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788188365Z namespaces/openstack-operators/deployments/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788258236Z namespaces/openstack-operators/imagestream/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788285397Z namespaces/openstack-operators/installplans/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78837992Z namespaces/openstack-operators/issuer/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788426301Z namespaces/openstack-operators/jobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788482782Z namespaces/openstack-operators/pods/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788592885Z namespaces/openstack-operators/pods/-describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788726789Z namespaces/openstack-operators/pods/logs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.78876156Z namespaces/openstack-operators/replicaset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788808001Z namespaces/openstack-operators/routes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788865552Z namespaces/openstack-operators/services/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788924744Z namespaces/openstack-operators/statefulset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.788964435Z namespaces/openstack-operators/subscriptions/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.789000496Z namespaces/openstack/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.789213782Z namespaces/openstack/all_resources.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.789457308Z namespaces/openstack/events.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.789736036Z namespaces/openstack/nad.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790011154Z namespaces/openstack/pvc.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790100396Z namespaces/openstack/buildconfig/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790174678Z namespaces/openstack/cert/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790202239Z namespaces/openstack/certificaterequests/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79026404Z namespaces/openstack/configmaps/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790316252Z namespaces/openstack/cronjobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790372553Z namespaces/openstack/daemonset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790422865Z namespaces/openstack/deployments/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790499087Z namespaces/openstack/imagestream/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790547998Z namespaces/openstack/installplans/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790584589Z namespaces/openstack/issuer/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79063685Z namespaces/openstack/jobs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790685702Z namespaces/openstack/pods/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790810635Z namespaces/openstack/pods/-describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790959Z namespaces/openstack/pods/logs/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.790999381Z namespaces/openstack/replicaset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791106964Z namespaces/openstack/routes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791189956Z namespaces/openstack/services/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791263468Z namespaces/openstack/statefulset/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791308159Z namespaces/openstack/subscriptions/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79134579Z network/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791478993Z network/l2advertisement.log [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791670759Z network/ipaddresspools/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791746531Z network/nnce/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791803302Z network/nncp/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791863454Z network/nns/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.791927475Z nodes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79209403Z nodes/crc.describe [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792401099Z nodes/crc.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792620895Z sos-reports/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792683296Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792764728Z sos-reports/_all_nodes/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79281059Z webhooks/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792855971Z webhooks/mutating/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.792885052Z webhooks/validating/ [must-gather-7q2jz] OUT 2025-11-26T11:09:31.793009355Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.793250861Z webhooks/validating/multus.openshift.io.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.793423756Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-7q2jz] OUT 2025-11-26T11:09:31.797879976Z [must-gather-7q2jz] OUT 2025-11-26T11:09:31.79796427Z sent 1,300 bytes received 26,553,341 bytes 17,703,094.00 bytes/sec [must-gather-7q2jz] OUT 2025-11-26T11:09:31.798046972Z total size is 183,572,248 speedup is 6.91 [must-gather ] OUT 2025-11-26T11:09:31.829156165Z namespace/openshift-must-gather-rbzfh deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing