[must-gather ] OUT 2025-10-02T17:53:37.733436556Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are updating, please see `oc get mcp` for further details clusteroperator/network is progressing: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" is not available (awaiting 1 nodes) DaemonSet "/openshift-multus/multus" is not available (awaiting 1 nodes) clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-02T17:53:37.817645377Z namespace/openshift-must-gather-r5j2t created [must-gather ] OUT 2025-10-02T17:53:37.827356873Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-v29sh created [must-gather ] OUT 2025-10-02T17:53:38.007987806Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-zskzz] OUT 2025-10-02T17:55:35.650929307Z waiting for gather to complete [must-gather-zskzz] OUT 2025-10-02T17:55:35.653424786Z downloading gather output [must-gather-zskzz] OUT 2025-10-02T17:55:35.959308207Z receiving incremental file list [must-gather-zskzz] OUT 2025-10-02T17:55:35.968374855Z ./ [must-gather-zskzz] OUT 2025-10-02T17:55:35.968724464Z must-gather.tar.xz [must-gather-zskzz] OUT 2025-10-02T17:55:36.062002306Z packagemanifests [must-gather-zskzz] OUT 2025-10-02T17:55:36.728272331Z run.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.7286348Z version [must-gather-zskzz] OUT 2025-10-02T17:55:36.728679812Z crd/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.728684782Z csv/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.728688812Z ctlplane/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729096424Z ctlplane/endpoints [must-gather-zskzz] OUT 2025-10-02T17:55:36.729299689Z ctlplane/services [must-gather-zskzz] OUT 2025-10-02T17:55:36.729461784Z ctlplane/ovn/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729485804Z ctlplane/rabbitmq/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729490334Z dbs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729494805Z namespaces/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729501675Z namespaces/openshift-machine-api/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.729921196Z namespaces/openshift-machine-api/all_resources.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.730169483Z namespaces/openshift-machine-api/events.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.730642206Z namespaces/openshift-machine-api/nad.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.730869462Z namespaces/openshift-machine-api/pvc.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.731023346Z namespaces/openshift-machine-api/buildconfig/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731034926Z namespaces/openshift-machine-api/cert/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731039046Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731043406Z namespaces/openshift-machine-api/configmaps/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731047296Z namespaces/openshift-machine-api/cronjobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731051017Z namespaces/openshift-machine-api/daemonset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731066267Z namespaces/openshift-machine-api/deployments/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.731667093Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.73189458Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.732075765Z namespaces/openshift-machine-api/imagestream/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.732085556Z namespaces/openshift-machine-api/installplans/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.732094176Z namespaces/openshift-machine-api/issuer/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.732098206Z namespaces/openshift-machine-api/jobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.732102036Z namespaces/openshift-machine-api/pods/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.732105896Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.7326301Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-zskzz] OUT 2025-10-02T17:55:36.732842416Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.733010371Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.733244267Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.733452683Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-zskzz] OUT 2025-10-02T17:55:36.733676869Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.733854073Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.734062329Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.734323316Z namespaces/openshift-machine-api/replicaset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.734334046Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736473965Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736490985Z namespaces/openshift-machine-api/routes/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736495196Z namespaces/openshift-machine-api/services/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736499576Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736503716Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736507486Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736511486Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736516696Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736520626Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.736524396Z namespaces/openshift-machine-api/statefulset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736528136Z namespaces/openshift-machine-api/subscriptions/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736531997Z namespaces/openstack-operators/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736535747Z namespaces/openstack-operators/all_resources.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.736539427Z namespaces/openstack-operators/events.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.736543847Z namespaces/openstack-operators/nad.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.736555287Z namespaces/openstack-operators/pvc.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.736559017Z namespaces/openstack-operators/buildconfig/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736582608Z namespaces/openstack-operators/cert/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736586368Z namespaces/openstack-operators/certificaterequests/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736590168Z namespaces/openstack-operators/configmaps/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736594038Z namespaces/openstack-operators/cronjobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736597688Z namespaces/openstack-operators/daemonset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736601438Z namespaces/openstack-operators/deployments/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736605379Z namespaces/openstack-operators/imagestream/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736609089Z namespaces/openstack-operators/installplans/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736612709Z namespaces/openstack-operators/issuer/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736616369Z namespaces/openstack-operators/jobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736620069Z namespaces/openstack-operators/pods/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736623709Z namespaces/openstack-operators/pods/-describe [must-gather-zskzz] OUT 2025-10-02T17:55:36.736627359Z namespaces/openstack-operators/pods/logs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736631029Z namespaces/openstack-operators/replicaset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.736635799Z namespaces/openstack-operators/routes/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73663943Z namespaces/openstack-operators/services/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73664309Z namespaces/openstack-operators/statefulset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73664676Z namespaces/openstack-operators/subscriptions/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73665047Z namespaces/openstack/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73665412Z namespaces/openstack/all_resources.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.73665779Z namespaces/openstack/events.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.73666148Z namespaces/openstack/nad.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.7366652Z namespaces/openstack/pvc.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.739141508Z namespaces/openstack/buildconfig/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739159459Z namespaces/openstack/cert/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739163579Z namespaces/openstack/certificaterequests/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739168509Z namespaces/openstack/configmaps/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739172299Z namespaces/openstack/cronjobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739176439Z namespaces/openstack/daemonset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739180159Z namespaces/openstack/deployments/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739183969Z namespaces/openstack/imagestream/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.739191279Z namespaces/openstack/installplans/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73919517Z namespaces/openstack/issuer/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73919892Z namespaces/openstack/jobs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.73920261Z namespaces/openstack/pods/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.74176533Z namespaces/openstack/pods/-describe [must-gather-zskzz] OUT 2025-10-02T17:55:36.741912054Z namespaces/openstack/pods/logs/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.741958005Z namespaces/openstack/replicaset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.741979326Z namespaces/openstack/routes/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742062528Z namespaces/openstack/services/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742093509Z namespaces/openstack/statefulset/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.74214342Z namespaces/openstack/subscriptions/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742167461Z network/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742307375Z network/l2advertisement.log [must-gather-zskzz] OUT 2025-10-02T17:55:36.742427828Z network/ipaddresspools/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742470849Z network/nnce/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.74249192Z network/nncp/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.74251711Z network/nns/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742557591Z nodes/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.742676015Z nodes/crc.describe [must-gather-zskzz] OUT 2025-10-02T17:55:36.743003604Z nodes/crc.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.743232841Z sos-reports/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.743300873Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-zskzz] OUT 2025-10-02T17:55:36.743340664Z sos-reports/_all_nodes/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.743367504Z webhooks/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.743408625Z webhooks/mutating/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.743436576Z webhooks/validating/ [must-gather-zskzz] OUT 2025-10-02T17:55:36.743537759Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.743720564Z webhooks/validating/multus.openshift.io.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.743882388Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-zskzz] OUT 2025-10-02T17:55:36.74799541Z [must-gather-zskzz] OUT 2025-10-02T17:55:36.748070052Z sent 1,129 bytes received 25,822,457 bytes 17,215,724.00 bytes/sec [must-gather-zskzz] OUT 2025-10-02T17:55:36.748115824Z total size is 175,574,258 speedup is 6.80 [must-gather ] OUT 2025-10-02T17:55:36.775865063Z namespace/openshift-must-gather-r5j2t deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing