[must-gather ] OUT 2025-10-10T00:18:03.869499699Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43348", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002090618), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-10T00:18:03.910148366Z namespace/openshift-must-gather-bvfww created [must-gather ] OUT 2025-10-10T00:18:03.914062397Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-5bfc9 created [must-gather ] OUT 2025-10-10T00:18:04.088660484Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-x76wt] OUT 2025-10-10T00:19:31.568798332Z waiting for gather to complete [must-gather-x76wt] OUT 2025-10-10T00:19:31.769488852Z downloading gather output [must-gather-x76wt] OUT 2025-10-10T00:19:32.48335386Z receiving incremental file list [must-gather-x76wt] OUT 2025-10-10T00:19:32.489487103Z ./ [must-gather-x76wt] OUT 2025-10-10T00:19:32.489754911Z must-gather.tar.xz [must-gather-x76wt] OUT 2025-10-10T00:19:32.557786923Z packagemanifests [must-gather-x76wt] OUT 2025-10-10T00:19:33.286907262Z run.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.286998424Z version [must-gather-x76wt] OUT 2025-10-10T00:19:33.287044945Z crd/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287050366Z csv/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287055176Z ctlplane/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287082667Z ctlplane/endpoints [must-gather-x76wt] OUT 2025-10-10T00:19:33.287127358Z ctlplane/services [must-gather-x76wt] OUT 2025-10-10T00:19:33.287186919Z ctlplane/ovn/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.28719755Z ctlplane/rabbitmq/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.28720757Z dbs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.28721822Z namespaces/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287228011Z namespaces/openshift-machine-api/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287290552Z namespaces/openshift-machine-api/all_resources.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.287307753Z namespaces/openshift-machine-api/events.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.287887659Z namespaces/openshift-machine-api/nad.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.28792331Z namespaces/openshift-machine-api/pvc.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.287964181Z namespaces/openshift-machine-api/buildconfig/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287972772Z namespaces/openshift-machine-api/cert/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287980402Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287987822Z namespaces/openshift-machine-api/configmaps/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.287995432Z namespaces/openshift-machine-api/cronjobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288002253Z namespaces/openshift-machine-api/daemonset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288009143Z namespaces/openshift-machine-api/deployments/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288015613Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288027683Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288034463Z namespaces/openshift-machine-api/imagestream/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288040774Z namespaces/openshift-machine-api/installplans/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288047104Z namespaces/openshift-machine-api/issuer/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288053414Z namespaces/openshift-machine-api/jobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288059774Z namespaces/openshift-machine-api/pods/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288066064Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288072894Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-x76wt] OUT 2025-10-10T00:19:33.288092005Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288102945Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.288110486Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288120446Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-x76wt] OUT 2025-10-10T00:19:33.288223429Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288241009Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.288375373Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.288440405Z namespaces/openshift-machine-api/replicaset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288454155Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288580149Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288593999Z namespaces/openshift-machine-api/routes/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.28860452Z namespaces/openshift-machine-api/services/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.288659541Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288774444Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.288877697Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.289241448Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.289389562Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.289448113Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.289497265Z namespaces/openshift-machine-api/statefulset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289505245Z namespaces/openshift-machine-api/subscriptions/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289512465Z namespaces/openstack-operators/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289565897Z namespaces/openstack-operators/all_resources.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.289745252Z namespaces/openstack-operators/events.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.289785483Z namespaces/openstack-operators/nad.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.289878706Z namespaces/openstack-operators/pvc.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.289912706Z namespaces/openstack-operators/buildconfig/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289930307Z namespaces/openstack-operators/cert/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289939657Z namespaces/openstack-operators/certificaterequests/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289949248Z namespaces/openstack-operators/configmaps/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289957998Z namespaces/openstack-operators/cronjobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289970638Z namespaces/openstack-operators/daemonset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289978928Z namespaces/openstack-operators/deployments/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289987189Z namespaces/openstack-operators/imagestream/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.289998019Z namespaces/openstack-operators/installplans/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290006619Z namespaces/openstack-operators/issuer/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290015119Z namespaces/openstack-operators/jobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29002352Z namespaces/openstack-operators/pods/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29004025Z namespaces/openstack-operators/pods/-describe [must-gather-x76wt] OUT 2025-10-10T00:19:33.290088681Z namespaces/openstack-operators/pods/logs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290097522Z namespaces/openstack-operators/replicaset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290106202Z namespaces/openstack-operators/routes/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290114892Z namespaces/openstack-operators/services/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290123852Z namespaces/openstack-operators/statefulset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290132453Z namespaces/openstack-operators/subscriptions/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290140923Z namespaces/openstack/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290150233Z namespaces/openstack/all_resources.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.290169194Z namespaces/openstack/events.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.290193414Z namespaces/openstack/nad.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.290279377Z namespaces/openstack/pvc.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.290312838Z namespaces/openstack/buildconfig/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290323888Z namespaces/openstack/cert/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290334468Z namespaces/openstack/certificaterequests/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290344989Z namespaces/openstack/configmaps/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290355179Z namespaces/openstack/cronjobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290365169Z namespaces/openstack/daemonset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29037638Z namespaces/openstack/deployments/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29038689Z namespaces/openstack/imagestream/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29039688Z namespaces/openstack/installplans/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.29040757Z namespaces/openstack/issuer/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290418181Z namespaces/openstack/jobs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290441881Z namespaces/openstack/pods/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290471352Z namespaces/openstack/pods/-describe [must-gather-x76wt] OUT 2025-10-10T00:19:33.290502933Z namespaces/openstack/pods/logs/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290513113Z namespaces/openstack/replicaset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290521894Z namespaces/openstack/routes/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290530144Z namespaces/openstack/services/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290539054Z namespaces/openstack/statefulset/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290547384Z namespaces/openstack/subscriptions/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290556115Z network/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290564495Z network/l2advertisement.log [must-gather-x76wt] OUT 2025-10-10T00:19:33.290573845Z network/ipaddresspools/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290582855Z network/nnce/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290591366Z network/nncp/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290599616Z network/nns/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290607906Z nodes/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290630477Z nodes/crc.describe [must-gather-x76wt] OUT 2025-10-10T00:19:33.290765701Z nodes/crc.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.290805582Z sos-reports/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290822112Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-x76wt] OUT 2025-10-10T00:19:33.290832622Z sos-reports/_all_nodes/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290842763Z webhooks/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290859983Z webhooks/mutating/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290870374Z webhooks/validating/ [must-gather-x76wt] OUT 2025-10-10T00:19:33.290886774Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.291032018Z webhooks/validating/multus.openshift.io.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.29109838Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-x76wt] OUT 2025-10-10T00:19:33.293785886Z [must-gather-x76wt] OUT 2025-10-10T00:19:33.293821177Z sent 1,129 bytes received 26,360,182 bytes 17,574,207.33 bytes/sec [must-gather-x76wt] OUT 2025-10-10T00:19:33.293832427Z total size is 177,640,976 speedup is 6.74 [must-gather ] OUT 2025-10-10T00:19:33.321134239Z namespace/openshift-must-gather-bvfww deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing