[must-gather ] OUT 2025-12-09T00:16:29.723366855Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.5 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42813", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0024391e8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-09T00:16:29.778962352Z namespace/openshift-must-gather-xxvtw created [must-gather ] OUT 2025-12-09T00:16:29.796777722Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-zbmv7 created [must-gather ] OUT 2025-12-09T00:16:29.807194687Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-4xs42] OUT 2025-12-09T00:18:05.904476136Z waiting for gather to complete [must-gather-4xs42] OUT 2025-12-09T00:18:05.907331261Z downloading gather output [must-gather-4xs42] OUT 2025-12-09T00:18:06.116333514Z receiving incremental file list [must-gather-4xs42] OUT 2025-12-09T00:18:06.126846632Z ./ [must-gather-4xs42] OUT 2025-12-09T00:18:06.126963605Z must-gather.tar.xz [must-gather-4xs42] OUT 2025-12-09T00:18:06.215538382Z packagemanifests [must-gather-4xs42] OUT 2025-12-09T00:18:07.015662111Z run.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.015796255Z version [must-gather-4xs42] OUT 2025-12-09T00:18:07.015803495Z crd/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.015807495Z csv/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.015811215Z ctlplane/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.015828436Z ctlplane/endpoints [must-gather-4xs42] OUT 2025-12-09T00:18:07.01598609Z ctlplane/services [must-gather-4xs42] OUT 2025-12-09T00:18:07.01600152Z ctlplane/ovn/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.0160066Z ctlplane/rabbitmq/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.01601083Z dbs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.01601622Z namespaces/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016020681Z namespaces/openshift-machine-api/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016076322Z namespaces/openshift-machine-api/all_resources.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.016220916Z namespaces/openshift-machine-api/events.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.016419272Z namespaces/openshift-machine-api/nad.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.016510614Z namespaces/openshift-machine-api/pvc.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.016543675Z namespaces/openshift-machine-api/buildconfig/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016557035Z namespaces/openshift-machine-api/cert/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016561116Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016568876Z namespaces/openshift-machine-api/configmaps/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016579716Z namespaces/openshift-machine-api/cronjobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016583926Z namespaces/openshift-machine-api/daemonset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016587806Z namespaces/openshift-machine-api/deployments/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016637168Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.016785891Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.016878214Z namespaces/openshift-machine-api/imagestream/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016900364Z namespaces/openshift-machine-api/installplans/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016905635Z namespaces/openshift-machine-api/issuer/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016909645Z namespaces/openshift-machine-api/jobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016915965Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016919775Z namespaces/openshift-machine-api/pods/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016923855Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.016963196Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-4xs42] OUT 2025-12-09T00:18:07.017094609Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.017161001Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.017288305Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.017363036Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-4xs42] OUT 2025-12-09T00:18:07.01748344Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.0175164Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.017620333Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.017720066Z namespaces/openshift-machine-api/replicaset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.017777407Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.0178708Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.017931791Z namespaces/openshift-machine-api/routes/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.017936031Z namespaces/openshift-machine-api/services/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.018021834Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.018144097Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.018230739Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.018335342Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.018531437Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.01864588Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.018725822Z namespaces/openshift-machine-api/statefulset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.018736502Z namespaces/openshift-machine-api/subscriptions/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.018740542Z namespaces/openstack-operators/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.018805834Z namespaces/openstack-operators/all_resources.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.018899576Z namespaces/openstack-operators/events.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.01902128Z namespaces/openstack-operators/nad.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.019135293Z namespaces/openstack-operators/pvc.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.019210776Z namespaces/openstack-operators/buildconfig/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019244266Z namespaces/openstack-operators/cert/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019267547Z namespaces/openstack-operators/certificaterequests/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019283877Z namespaces/openstack-operators/configmaps/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019299948Z namespaces/openstack-operators/cronjobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019315728Z namespaces/openstack-operators/daemonset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019331309Z namespaces/openstack-operators/deployments/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019347749Z namespaces/openstack-operators/imagestream/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.01936751Z namespaces/openstack-operators/installplans/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.01939538Z namespaces/openstack-operators/issuer/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019403391Z namespaces/openstack-operators/jobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019430661Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.019435301Z namespaces/openstack-operators/pods/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.01977326Z namespaces/openstack-operators/pods/-describe [must-gather-4xs42] OUT 2025-12-09T00:18:07.020070278Z namespaces/openstack-operators/pods/logs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.02015553Z namespaces/openstack-operators/replicaset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.02016635Z namespaces/openstack-operators/routes/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.020170591Z namespaces/openstack-operators/services/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.020174901Z namespaces/openstack-operators/statefulset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.020178801Z namespaces/openstack-operators/subscriptions/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.020182631Z namespaces/openstack/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.020363976Z namespaces/openstack/all_resources.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.02053455Z namespaces/openstack/events.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.020713785Z namespaces/openstack/nad.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.020978862Z namespaces/openstack/pvc.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.021057804Z namespaces/openstack/buildconfig/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021063394Z namespaces/openstack/cert/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021067834Z namespaces/openstack/certificaterequests/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021072564Z namespaces/openstack/configmaps/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021077234Z namespaces/openstack/cronjobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021085104Z namespaces/openstack/daemonset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021089904Z namespaces/openstack/deployments/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021094815Z namespaces/openstack/imagestream/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021099465Z namespaces/openstack/installplans/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021110995Z namespaces/openstack/issuer/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021115605Z namespaces/openstack/jobs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021123165Z namespaces/openstack/poddisruptionbudgets/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021130596Z namespaces/openstack/pods/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021249369Z namespaces/openstack/pods/-describe [must-gather-4xs42] OUT 2025-12-09T00:18:07.021378782Z namespaces/openstack/pods/logs/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021384712Z namespaces/openstack/replicaset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021389092Z namespaces/openstack/routes/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021393852Z namespaces/openstack/services/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021399213Z namespaces/openstack/statefulset/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021404383Z namespaces/openstack/subscriptions/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021409193Z network/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021564517Z network/l2advertisement.log [must-gather-4xs42] OUT 2025-12-09T00:18:07.02168024Z network/ipaddresspools/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.02168572Z network/nnce/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.02169012Z network/nncp/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021712941Z network/nns/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021726821Z nodes/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.021901606Z nodes/crc.describe [must-gather-4xs42] OUT 2025-12-09T00:18:07.0224158Z nodes/crc.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.022678367Z sos-reports/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.022692037Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-4xs42] OUT 2025-12-09T00:18:07.022759149Z sos-reports/_all_nodes/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.022766839Z webhooks/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.022771029Z webhooks/mutating/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.02281346Z webhooks/validating/ [must-gather-4xs42] OUT 2025-12-09T00:18:07.022963774Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.023237241Z webhooks/validating/multus.openshift.io.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.023537559Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-4xs42] OUT 2025-12-09T00:18:07.029368173Z [must-gather-4xs42] OUT 2025-12-09T00:18:07.029400004Z sent 1,141 bytes received 27,516,607 bytes 11,007,099.20 bytes/sec [must-gather-4xs42] OUT 2025-12-09T00:18:07.029408204Z total size is 187,038,007 speedup is 6.80 [must-gather ] OUT 2025-12-09T00:18:07.062243141Z namespace/openshift-must-gather-xxvtw deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.5 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing