[must-gather ] OUT 2025-12-02T00:17:06.901640554Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42846", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002ea3f38), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-02T00:17:06.959934635Z namespace/openshift-must-gather-c2bkv created [must-gather ] OUT 2025-12-02T00:17:06.982459208Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-2b55p created [must-gather ] OUT 2025-12-02T00:17:07.00193702Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-mvzg7] OUT 2025-12-02T00:18:56.337749911Z waiting for gather to complete [must-gather-mvzg7] OUT 2025-12-02T00:18:56.345391595Z downloading gather output [must-gather-mvzg7] OUT 2025-12-02T00:18:56.659990163Z receiving incremental file list [must-gather-mvzg7] OUT 2025-12-02T00:18:56.70226778Z ./ [must-gather-mvzg7] OUT 2025-12-02T00:18:56.702435134Z must-gather.tar.xz [must-gather-mvzg7] OUT 2025-12-02T00:18:56.794226962Z packagemanifests [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499756535Z run.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499830357Z version [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499841107Z crd/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499849687Z csv/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499857817Z ctlplane/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.499871328Z ctlplane/endpoints [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500127344Z ctlplane/services [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500165375Z ctlplane/ovn/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500183995Z ctlplane/rabbitmq/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500188625Z dbs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500217886Z namespaces/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500222216Z namespaces/openshift-machine-api/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500232166Z namespaces/openshift-machine-api/all_resources.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500518284Z namespaces/openshift-machine-api/events.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50078316Z namespaces/openshift-machine-api/nad.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500934974Z namespaces/openshift-machine-api/pvc.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.500977695Z namespaces/openshift-machine-api/buildconfig/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501030886Z namespaces/openshift-machine-api/cert/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501037046Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501164259Z namespaces/openshift-machine-api/configmaps/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501176019Z namespaces/openshift-machine-api/cronjobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501180239Z namespaces/openshift-machine-api/daemonset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501184999Z namespaces/openshift-machine-api/deployments/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501263331Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501542938Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501669851Z namespaces/openshift-machine-api/imagestream/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501677881Z namespaces/openshift-machine-api/installplans/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501682011Z namespaces/openshift-machine-api/issuer/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501686281Z namespaces/openshift-machine-api/jobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501692761Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501697381Z namespaces/openshift-machine-api/pods/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501705372Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.501841886Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50201139Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.502137793Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.502333267Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50246354Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-mvzg7] OUT 2025-12-02T00:18:57.502636054Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.502762657Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.502963782Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503130916Z namespaces/openshift-machine-api/replicaset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50324573Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503452794Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503591298Z namespaces/openshift-machine-api/routes/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503598008Z namespaces/openshift-machine-api/services/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503715921Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.503905465Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.504084149Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.504263633Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.504608042Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.504785616Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.504907339Z namespaces/openshift-machine-api/statefulset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50491502Z namespaces/openshift-machine-api/subscriptions/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50491944Z namespaces/openstack-operators/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505082623Z namespaces/openstack-operators/all_resources.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505239737Z namespaces/openstack-operators/events.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505441102Z namespaces/openstack-operators/nad.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505613626Z namespaces/openstack-operators/pvc.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505669297Z namespaces/openstack-operators/buildconfig/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505675367Z namespaces/openstack-operators/cert/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505679767Z namespaces/openstack-operators/certificaterequests/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505684087Z namespaces/openstack-operators/configmaps/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505688008Z namespaces/openstack-operators/cronjobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505706898Z namespaces/openstack-operators/daemonset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505713388Z namespaces/openstack-operators/deployments/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505717858Z namespaces/openstack-operators/imagestream/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505721798Z namespaces/openstack-operators/installplans/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505728178Z namespaces/openstack-operators/issuer/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505732339Z namespaces/openstack-operators/jobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505736719Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505741219Z namespaces/openstack-operators/pods/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.505946104Z namespaces/openstack-operators/pods/-describe [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506083378Z namespaces/openstack-operators/pods/logs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506092268Z namespaces/openstack-operators/replicaset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506096888Z namespaces/openstack-operators/routes/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506100688Z namespaces/openstack-operators/services/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506104868Z namespaces/openstack-operators/statefulset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506108598Z namespaces/openstack-operators/subscriptions/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506118149Z namespaces/openstack/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50617836Z namespaces/openstack/all_resources.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506417215Z namespaces/openstack/events.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506578219Z namespaces/openstack/nad.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506949038Z namespaces/openstack/pvc.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506965358Z namespaces/openstack/buildconfig/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506971288Z namespaces/openstack/cert/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506976979Z namespaces/openstack/certificaterequests/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506982379Z namespaces/openstack/configmaps/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506991109Z namespaces/openstack/cronjobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.506996449Z namespaces/openstack/daemonset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507001399Z namespaces/openstack/deployments/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507006569Z namespaces/openstack/imagestream/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507012069Z namespaces/openstack/installplans/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507032Z namespaces/openstack/issuer/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50703718Z namespaces/openstack/jobs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.5070426Z namespaces/openstack/poddisruptionbudgets/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50704789Z namespaces/openstack/pods/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507298136Z namespaces/openstack/pods/-describe [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507341697Z namespaces/openstack/pods/logs/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507354157Z namespaces/openstack/replicaset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507360608Z namespaces/openstack/routes/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507364398Z namespaces/openstack/services/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507368738Z namespaces/openstack/statefulset/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507372828Z namespaces/openstack/subscriptions/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507376808Z network/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.50741036Z network/l2advertisement.log [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507597904Z network/ipaddresspools/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507609754Z network/nnce/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507616755Z network/nncp/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507622245Z network/nns/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507627415Z nodes/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507642265Z nodes/crc.describe [must-gather-mvzg7] OUT 2025-12-02T00:18:57.507990613Z nodes/crc.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508431234Z sos-reports/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508458854Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508468474Z sos-reports/_all_nodes/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508477215Z webhooks/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508486095Z webhooks/mutating/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508494675Z webhooks/validating/ [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508502845Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508511485Z webhooks/validating/multus.openshift.io.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.508526226Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-mvzg7] OUT 2025-12-02T00:18:57.512370358Z [must-gather-mvzg7] OUT 2025-12-02T00:18:57.512403699Z sent 1,141 bytes received 27,090,458 bytes 18,061,066.00 bytes/sec [must-gather-mvzg7] OUT 2025-12-02T00:18:57.51241474Z total size is 185,099,251 speedup is 6.83 [must-gather ] OUT 2025-12-02T00:18:57.541887049Z namespace/openshift-must-gather-c2bkv deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing