[must-gather ] OUT 2025-10-01T18:48:43.838264664Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-01T18:48:43.901502568Z namespace/openshift-must-gather-sd27w created [must-gather ] OUT 2025-10-01T18:48:43.905544514Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-t2wbq created [must-gather ] OUT 2025-10-01T18:48:44.039653194Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-8tttg] OUT 2025-10-01T18:50:40.749616983Z waiting for gather to complete [must-gather-8tttg] OUT 2025-10-01T18:50:40.752486481Z downloading gather output [must-gather-8tttg] OUT 2025-10-01T18:50:41.034484719Z receiving incremental file list [must-gather-8tttg] OUT 2025-10-01T18:50:41.044418059Z ./ [must-gather-8tttg] OUT 2025-10-01T18:50:41.044659936Z must-gather.tar.xz [must-gather-8tttg] OUT 2025-10-01T18:50:41.103565216Z packagemanifests [must-gather-8tttg] OUT 2025-10-01T18:50:41.834453845Z run.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.834688081Z version [must-gather-8tttg] OUT 2025-10-01T18:50:41.834952859Z crd/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.834964049Z csv/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.834968139Z ctlplane/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835187745Z ctlplane/endpoints [must-gather-8tttg] OUT 2025-10-01T18:50:41.835692779Z ctlplane/services [must-gather-8tttg] OUT 2025-10-01T18:50:41.835762321Z ctlplane/ovn/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835801232Z ctlplane/rabbitmq/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835818482Z dbs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835841843Z namespaces/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835866283Z namespaces/cert-manager/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.835964926Z namespaces/cert-manager/all_resources.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.836276935Z namespaces/cert-manager/events.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.83647495Z namespaces/cert-manager/nad.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.836667905Z namespaces/cert-manager/pvc.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.836821019Z namespaces/cert-manager/buildconfig/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83683139Z namespaces/cert-manager/cert/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83683669Z namespaces/cert-manager/certificaterequests/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83684172Z namespaces/cert-manager/configmaps/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83684651Z namespaces/cert-manager/cronjobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83685123Z namespaces/cert-manager/daemonset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.83685604Z namespaces/cert-manager/deployments/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837621821Z namespaces/cert-manager/deployments/cert-manager-cainjector.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.837634191Z namespaces/cert-manager/deployments/cert-manager-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.837639392Z namespaces/cert-manager/deployments/cert-manager.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.837644302Z namespaces/cert-manager/imagestream/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837649842Z namespaces/cert-manager/installplans/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837654872Z namespaces/cert-manager/issuer/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837668692Z namespaces/cert-manager/jobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837673493Z namespaces/cert-manager/pods/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837678003Z namespaces/cert-manager/pods/cert-manager-67c98b89c8-9gk5w/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.837682823Z namespaces/cert-manager/pods/cert-manager-67c98b89c8-9gk5w/cert-manager-67c98b89c8-9gk5w-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.838863125Z namespaces/cert-manager/pods/cert-manager-67c98b89c8-9gk5w/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.838876325Z namespaces/cert-manager/pods/cert-manager-67c98b89c8-9gk5w/logs/cert-manager-controller.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.838881595Z namespaces/cert-manager/pods/cert-manager-cainjector-5c5695d979-nq2z9/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.838886675Z namespaces/cert-manager/pods/cert-manager-cainjector-5c5695d979-nq2z9/cert-manager-cainjector-5c5695d979-nq2z9-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.838891696Z namespaces/cert-manager/pods/cert-manager-cainjector-5c5695d979-nq2z9/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.838896736Z namespaces/cert-manager/pods/cert-manager-cainjector-5c5695d979-nq2z9/logs/cert-manager-cainjector.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.838901506Z namespaces/cert-manager/pods/cert-manager-webhook-7f9f8648b9-gnkdf/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.838906286Z namespaces/cert-manager/pods/cert-manager-webhook-7f9f8648b9-gnkdf/cert-manager-webhook-7f9f8648b9-gnkdf-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.838921286Z namespaces/cert-manager/pods/cert-manager-webhook-7f9f8648b9-gnkdf/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.838993818Z namespaces/cert-manager/pods/cert-manager-webhook-7f9f8648b9-gnkdf/logs/cert-manager-webhook.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.839161553Z namespaces/cert-manager/replicaset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.839377649Z namespaces/cert-manager/replicaset/cert-manager-67c98b89c8.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.839712088Z namespaces/cert-manager/replicaset/cert-manager-cainjector-5c5695d979.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.840055307Z namespaces/cert-manager/replicaset/cert-manager-webhook-7f9f8648b9.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.840272213Z namespaces/cert-manager/routes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.840329135Z namespaces/cert-manager/services/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.840475759Z namespaces/cert-manager/services/cert-manager-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.840765996Z namespaces/cert-manager/services/cert-manager.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.840929281Z namespaces/cert-manager/statefulset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.840961792Z namespaces/cert-manager/subscriptions/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.841023323Z namespaces/openshift-machine-api/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.841225649Z namespaces/openshift-machine-api/all_resources.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.841412844Z namespaces/openshift-machine-api/events.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.841739673Z namespaces/openshift-machine-api/nad.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.841921418Z namespaces/openshift-machine-api/pvc.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.842142674Z namespaces/openshift-machine-api/buildconfig/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842152284Z namespaces/openshift-machine-api/cert/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842156204Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842159924Z namespaces/openshift-machine-api/configmaps/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842163614Z namespaces/openshift-machine-api/cronjobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842167315Z namespaces/openshift-machine-api/daemonset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842340609Z namespaces/openshift-machine-api/deployments/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.84235289Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.842567555Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.842777831Z namespaces/openshift-machine-api/imagestream/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842828803Z namespaces/openshift-machine-api/installplans/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842852203Z namespaces/openshift-machine-api/issuer/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.842926245Z namespaces/openshift-machine-api/jobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.843209483Z namespaces/openshift-machine-api/pods/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.843288945Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.84347355Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.843771928Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.843927592Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.84420958Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.844408215Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.84456603Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.844752365Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.84494119Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.845115755Z namespaces/openshift-machine-api/replicaset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.845270249Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.845448384Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.845586747Z namespaces/openshift-machine-api/routes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.845630629Z namespaces/openshift-machine-api/services/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.845770852Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.845951257Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.846124602Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.846297427Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.846536483Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.846752489Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.846876412Z namespaces/openshift-machine-api/statefulset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.846921804Z namespaces/openshift-machine-api/subscriptions/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.846944154Z namespaces/openstack-operators/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.847081328Z namespaces/openstack-operators/all_resources.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.847261063Z namespaces/openstack-operators/events.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.847435578Z namespaces/openstack-operators/nad.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.847672404Z namespaces/openstack-operators/pvc.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.847796027Z namespaces/openstack-operators/buildconfig/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.847844079Z namespaces/openstack-operators/cert/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.84788076Z namespaces/openstack-operators/certificaterequests/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.847915001Z namespaces/openstack-operators/configmaps/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.847970192Z namespaces/openstack-operators/cronjobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848024724Z namespaces/openstack-operators/daemonset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848061425Z namespaces/openstack-operators/deployments/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848119306Z namespaces/openstack-operators/imagestream/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848175918Z namespaces/openstack-operators/installplans/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848204798Z namespaces/openstack-operators/issuer/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848286221Z namespaces/openstack-operators/jobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848323972Z namespaces/openstack-operators/pods/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848454655Z namespaces/openstack-operators/pods/-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.848599869Z namespaces/openstack-operators/pods/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848653771Z namespaces/openstack-operators/replicaset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848684522Z namespaces/openstack-operators/routes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848750803Z namespaces/openstack-operators/services/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848800195Z namespaces/openstack-operators/statefulset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848851296Z namespaces/openstack-operators/subscriptions/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.848894567Z namespaces/openstack/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849078122Z namespaces/openstack/all_resources.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.849228196Z namespaces/openstack/events.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.8493729Z namespaces/openstack/nad.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.849512644Z namespaces/openstack/pvc.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.849591256Z namespaces/openstack/buildconfig/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849631347Z namespaces/openstack/cert/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849668798Z namespaces/openstack/certificaterequests/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849693549Z namespaces/openstack/configmaps/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.84973462Z namespaces/openstack/cronjobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849784371Z namespaces/openstack/daemonset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849844963Z namespaces/openstack/deployments/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849872954Z namespaces/openstack/imagestream/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849904745Z namespaces/openstack/installplans/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.849960076Z namespaces/openstack/issuer/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850055619Z namespaces/openstack/jobs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.85009379Z namespaces/openstack/pods/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850240744Z namespaces/openstack/pods/-describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.850386428Z namespaces/openstack/pods/logs/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850431479Z namespaces/openstack/replicaset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.85045297Z namespaces/openstack/routes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850506511Z namespaces/openstack/services/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850539612Z namespaces/openstack/statefulset/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850572693Z namespaces/openstack/subscriptions/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850632174Z network/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850798079Z network/l2advertisement.log [must-gather-8tttg] OUT 2025-10-01T18:50:41.850923952Z network/ipaddresspools/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.850963093Z network/nnce/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851011515Z network/nncp/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851042676Z network/nns/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851079006Z nodes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.85119258Z nodes/crc.describe [must-gather-8tttg] OUT 2025-10-01T18:50:41.851475337Z nodes/crc.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.851700843Z sos-reports/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851769025Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-8tttg] OUT 2025-10-01T18:50:41.851831317Z sos-reports/_all_nodes/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851878328Z webhooks/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.851910309Z webhooks/mutating/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.852041273Z webhooks/mutating/cert-manager-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.852202867Z webhooks/validating/ [must-gather-8tttg] OUT 2025-10-01T18:50:41.852335321Z webhooks/validating/cert-manager-webhook.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.852544056Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.852736692Z webhooks/validating/multus.openshift.io.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.852929687Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-8tttg] OUT 2025-10-01T18:50:41.857259584Z [must-gather-8tttg] OUT 2025-10-01T18:50:41.857366127Z sent 1,605 bytes received 25,819,401 bytes 17,214,004.00 bytes/sec [must-gather-8tttg] OUT 2025-10-01T18:50:41.857412469Z total size is 175,165,876 speedup is 6.78 [must-gather ] OUT 2025-10-01T18:50:41.886600741Z namespace/openshift-must-gather-sd27w deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing