[must-gather ] OUT 2025-10-08T00:23:45.635582294Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42952", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003b40ab0), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-08T00:23:45.685673456Z namespace/openshift-must-gather-v4ptb created [must-gather ] OUT 2025-10-08T00:23:45.748699087Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-588qw created [must-gather ] OUT 2025-10-08T00:23:45.916478363Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-76ww8] OUT 2025-10-08T00:25:40.552024624Z waiting for gather to complete [must-gather-76ww8] OUT 2025-10-08T00:25:40.554164642Z downloading gather output [must-gather-76ww8] OUT 2025-10-08T00:25:40.796803224Z receiving incremental file list [must-gather-76ww8] OUT 2025-10-08T00:25:40.802762013Z ./ [must-gather-76ww8] OUT 2025-10-08T00:25:40.802904317Z must-gather.tar.xz [must-gather-76ww8] OUT 2025-10-08T00:25:40.848898605Z packagemanifests [must-gather-76ww8] OUT 2025-10-08T00:25:41.500827265Z run.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.500949918Z version [must-gather-76ww8] OUT 2025-10-08T00:25:41.50098463Z crd/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50099549Z csv/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50100209Z ctlplane/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501036801Z ctlplane/endpoints [must-gather-76ww8] OUT 2025-10-08T00:25:41.501103003Z ctlplane/services [must-gather-76ww8] OUT 2025-10-08T00:25:41.501160615Z ctlplane/ovn/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501169575Z ctlplane/rabbitmq/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501177016Z dbs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501181356Z namespaces/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501186196Z namespaces/openshift-machine-api/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501258758Z namespaces/openshift-machine-api/all_resources.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.501460125Z namespaces/openshift-machine-api/events.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.501656001Z namespaces/openshift-machine-api/nad.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.501779945Z namespaces/openshift-machine-api/pvc.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.501834867Z namespaces/openshift-machine-api/buildconfig/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501840197Z namespaces/openshift-machine-api/cert/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501844817Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501851597Z namespaces/openshift-machine-api/configmaps/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501855967Z namespaces/openshift-machine-api/cronjobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501859928Z namespaces/openshift-machine-api/daemonset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501864108Z namespaces/openshift-machine-api/deployments/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.501962301Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.502118596Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.502207889Z namespaces/openshift-machine-api/imagestream/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502218749Z namespaces/openshift-machine-api/installplans/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502223819Z namespaces/openshift-machine-api/issuer/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502228769Z namespaces/openshift-machine-api/jobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50223343Z namespaces/openshift-machine-api/pods/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50223856Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502296622Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-76ww8] OUT 2025-10-08T00:25:41.502452237Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50254795Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.502683084Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502750656Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-76ww8] OUT 2025-10-08T00:25:41.502901551Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.502965723Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.503117998Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.503300674Z namespaces/openshift-machine-api/replicaset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.503359155Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.50350005Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.503577222Z namespaces/openshift-machine-api/routes/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.503582963Z namespaces/openshift-machine-api/services/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.503645915Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.503803659Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.503909422Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.504036096Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.504231992Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.504363227Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.504434639Z namespaces/openshift-machine-api/statefulset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504441239Z namespaces/openshift-machine-api/subscriptions/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504445679Z namespaces/openstack-operators/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504501391Z namespaces/openstack-operators/all_resources.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.504619485Z namespaces/openstack-operators/events.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.504706318Z namespaces/openstack-operators/nad.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.504836252Z namespaces/openstack-operators/pvc.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.504879143Z namespaces/openstack-operators/buildconfig/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504884153Z namespaces/openstack-operators/cert/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504888273Z namespaces/openstack-operators/certificaterequests/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504892194Z namespaces/openstack-operators/configmaps/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504896424Z namespaces/openstack-operators/cronjobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504900684Z namespaces/openstack-operators/daemonset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504910254Z namespaces/openstack-operators/deployments/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504917744Z namespaces/openstack-operators/imagestream/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504921794Z namespaces/openstack-operators/installplans/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504925695Z namespaces/openstack-operators/issuer/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504929745Z namespaces/openstack-operators/jobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504933565Z namespaces/openstack-operators/pods/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.504982526Z namespaces/openstack-operators/pods/-describe [must-gather-76ww8] OUT 2025-10-08T00:25:41.505059249Z namespaces/openstack-operators/pods/logs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505064779Z namespaces/openstack-operators/replicaset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505068999Z namespaces/openstack-operators/routes/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505072659Z namespaces/openstack-operators/services/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505076959Z namespaces/openstack-operators/statefulset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50508508Z namespaces/openstack-operators/subscriptions/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.50508936Z namespaces/openstack/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505137181Z namespaces/openstack/all_resources.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.505249615Z namespaces/openstack/events.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.50540788Z namespaces/openstack/nad.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.505534844Z namespaces/openstack/pvc.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.505576465Z namespaces/openstack/buildconfig/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505581176Z namespaces/openstack/cert/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505585006Z namespaces/openstack/certificaterequests/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505588846Z namespaces/openstack/configmaps/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505592976Z namespaces/openstack/cronjobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505596676Z namespaces/openstack/daemonset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505610957Z namespaces/openstack/deployments/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505615007Z namespaces/openstack/imagestream/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505619397Z namespaces/openstack/installplans/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505623177Z namespaces/openstack/issuer/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505627107Z namespaces/openstack/jobs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505630817Z namespaces/openstack/pods/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505693529Z namespaces/openstack/pods/-describe [must-gather-76ww8] OUT 2025-10-08T00:25:41.505791552Z namespaces/openstack/pods/logs/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505805873Z namespaces/openstack/replicaset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505811133Z namespaces/openstack/routes/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505815983Z namespaces/openstack/services/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505820403Z namespaces/openstack/statefulset/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505824613Z namespaces/openstack/subscriptions/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505831144Z network/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505861695Z network/l2advertisement.log [must-gather-76ww8] OUT 2025-10-08T00:25:41.505947707Z network/ipaddresspools/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505959778Z network/nnce/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505964338Z network/nncp/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505968988Z network/nns/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.505973598Z nodes/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506008889Z nodes/crc.describe [must-gather-76ww8] OUT 2025-10-08T00:25:41.506252127Z nodes/crc.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.506385241Z sos-reports/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506402982Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-76ww8] OUT 2025-10-08T00:25:41.506408392Z sos-reports/_all_nodes/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506413062Z webhooks/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506417692Z webhooks/mutating/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506422082Z webhooks/validating/ [must-gather-76ww8] OUT 2025-10-08T00:25:41.506478093Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.506666909Z webhooks/validating/multus.openshift.io.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.506777393Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-76ww8] OUT 2025-10-08T00:25:41.509448267Z [must-gather-76ww8] OUT 2025-10-08T00:25:41.509499639Z sent 1,129 bytes received 24,890,959 bytes 16,594,725.33 bytes/sec [must-gather-76ww8] OUT 2025-10-08T00:25:41.509507379Z total size is 167,896,445 speedup is 6.74 [must-gather ] OUT 2025-10-08T00:25:41.536471123Z namespace/openshift-must-gather-v4ptb deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing