[must-gather ] OUT 2025-10-06T00:27:33.750622462Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42708", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003fd2e10), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-06T00:27:33.793874868Z namespace/openshift-must-gather-6xdw8 created [must-gather ] OUT 2025-10-06T00:27:33.797998812Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-wdqxp created [must-gather ] OUT 2025-10-06T00:27:33.924824436Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-vtxrt] OUT 2025-10-06T00:29:07.089044176Z waiting for gather to complete [must-gather-vtxrt] OUT 2025-10-06T00:29:07.092377148Z downloading gather output [must-gather-vtxrt] OUT 2025-10-06T00:29:07.414975403Z receiving incremental file list [must-gather-vtxrt] OUT 2025-10-06T00:29:07.425494254Z ./ [must-gather-vtxrt] OUT 2025-10-06T00:29:07.425628887Z must-gather.tar.xz [must-gather-vtxrt] OUT 2025-10-06T00:29:07.500146367Z packagemanifests [must-gather-vtxrt] OUT 2025-10-06T00:29:08.215887559Z run.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.2159418Z version [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216052163Z crd/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216068514Z csv/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216073174Z ctlplane/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216078304Z ctlplane/endpoints [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216085234Z ctlplane/services [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216158826Z ctlplane/ovn/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216164666Z ctlplane/rabbitmq/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216168567Z dbs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216172357Z namespaces/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216176367Z namespaces/openshift-machine-api/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216248479Z namespaces/openshift-machine-api/all_resources.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216381872Z namespaces/openshift-machine-api/events.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216562087Z namespaces/openshift-machine-api/nad.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21665291Z namespaces/openshift-machine-api/pvc.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216693731Z namespaces/openshift-machine-api/buildconfig/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216700841Z namespaces/openshift-machine-api/cert/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216706851Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216712422Z namespaces/openshift-machine-api/configmaps/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216718072Z namespaces/openshift-machine-api/cronjobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216723942Z namespaces/openshift-machine-api/daemonset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216729582Z namespaces/openshift-machine-api/deployments/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216739052Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216876516Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216921027Z namespaces/openshift-machine-api/imagestream/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216958998Z namespaces/openshift-machine-api/installplans/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216978099Z namespaces/openshift-machine-api/issuer/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216987639Z namespaces/openshift-machine-api/jobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216993409Z namespaces/openshift-machine-api/pods/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.216999219Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217059241Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217101852Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217176604Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217289137Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217321968Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217497013Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217518804Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217618586Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217681858Z namespaces/openshift-machine-api/replicaset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217765581Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217835182Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217897714Z namespaces/openshift-machine-api/routes/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217921025Z namespaces/openshift-machine-api/services/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.217985247Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.2180958Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218217453Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218350277Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218556692Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218649775Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218720797Z namespaces/openshift-machine-api/statefulset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218726797Z namespaces/openshift-machine-api/subscriptions/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218731187Z namespaces/openstack-operators/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.2188211Z namespaces/openstack-operators/all_resources.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218882531Z namespaces/openstack-operators/events.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.218956313Z namespaces/openstack-operators/nad.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219063306Z namespaces/openstack-operators/pvc.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219093807Z namespaces/openstack-operators/buildconfig/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219099607Z namespaces/openstack-operators/cert/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219104718Z namespaces/openstack-operators/certificaterequests/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219110438Z namespaces/openstack-operators/configmaps/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219115818Z namespaces/openstack-operators/cronjobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219120988Z namespaces/openstack-operators/daemonset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219134248Z namespaces/openstack-operators/deployments/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219139529Z namespaces/openstack-operators/imagestream/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219144259Z namespaces/openstack-operators/installplans/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219149349Z namespaces/openstack-operators/issuer/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219153889Z namespaces/openstack-operators/jobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219158869Z namespaces/openstack-operators/pods/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219170689Z namespaces/openstack-operators/pods/-describe [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21919151Z namespaces/openstack-operators/pods/logs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.2192Z namespaces/openstack-operators/replicaset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21920694Z namespaces/openstack-operators/routes/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219232851Z namespaces/openstack-operators/services/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219239941Z namespaces/openstack-operators/statefulset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219247242Z namespaces/openstack-operators/subscriptions/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219504439Z namespaces/openstack/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21953735Z namespaces/openstack/all_resources.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21954818Z namespaces/openstack/events.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219622702Z namespaces/openstack/nad.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219721345Z namespaces/openstack/pvc.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219731425Z namespaces/openstack/buildconfig/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219736475Z namespaces/openstack/cert/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219741165Z namespaces/openstack/certificaterequests/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219746275Z namespaces/openstack/configmaps/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219750795Z namespaces/openstack/cronjobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219755296Z namespaces/openstack/daemonset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219760856Z namespaces/openstack/deployments/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219770666Z namespaces/openstack/imagestream/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219780766Z namespaces/openstack/installplans/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219784826Z namespaces/openstack/issuer/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219788717Z namespaces/openstack/jobs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219793327Z namespaces/openstack/pods/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219819677Z namespaces/openstack/pods/-describe [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219842748Z namespaces/openstack/pods/logs/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219855448Z namespaces/openstack/replicaset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219862619Z namespaces/openstack/routes/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219888699Z namespaces/openstack/services/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21990213Z namespaces/openstack/statefulset/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21990978Z namespaces/openstack/subscriptions/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21991766Z network/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.21992593Z network/l2advertisement.log [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219960371Z network/ipaddresspools/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219967881Z network/nnce/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219973782Z network/nncp/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219979662Z network/nns/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.219985692Z nodes/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220026373Z nodes/crc.describe [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220248779Z nodes/crc.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220348592Z sos-reports/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220355002Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220362262Z sos-reports/_all_nodes/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220366542Z webhooks/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220374313Z webhooks/mutating/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220379293Z webhooks/validating/ [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220422244Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220551018Z webhooks/validating/multus.openshift.io.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.220775144Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-vtxrt] OUT 2025-10-06T00:29:08.2231647Z [must-gather-vtxrt] OUT 2025-10-06T00:29:08.22318206Z sent 1,129 bytes received 25,221,318 bytes 16,814,964.67 bytes/sec [must-gather-vtxrt] OUT 2025-10-06T00:29:08.22318733Z total size is 174,757,347 speedup is 6.93 [must-gather ] OUT 2025-10-06T00:29:08.256292375Z namespace/openshift-must-gather-6xdw8 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is degraded because WellKnownReadyControllerDegraded: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42708", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003fd2e10), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly) clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing