[must-gather ] OUT 2025-10-01T07:31:45.069292343Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-01T07:31:45.133668917Z namespace/openshift-must-gather-6nk2b created [must-gather ] OUT 2025-10-01T07:31:45.139822873Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-2jbcp created [must-gather ] OUT 2025-10-01T07:31:45.295208253Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-mzhj8] POD 2025-10-01T07:31:52.559037625Z volume percentage checker started..... [must-gather-mzhj8] POD 2025-10-01T07:31:52.579263865Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:31:52.764925732Z Error from server (NotFound): namespaces "openstack" not found [must-gather-mzhj8] POD 2025-10-01T07:31:52.944533495Z Will retrieve SOS reports from nodes [must-gather-mzhj8] POD 2025-10-01T07:31:53.132456030Z error: the server doesn't have a resource type "openstackdataplanenodesets" [must-gather-mzhj8] POD 2025-10-01T07:31:53.994578400Z Gathering CRDs [must-gather-mzhj8] POD 2025-10-01T07:31:54.485781432Z Gathering CRs [must-gather-mzhj8] POD 2025-10-01T07:31:54.652747280Z Writing validating webhooks [must-gather-mzhj8] POD 2025-10-01T07:31:54.835925794Z Writing mutating webhooks [must-gather-mzhj8] POD 2025-10-01T07:31:56.069708397Z Gathering secrets in namespace openstack [must-gather-mzhj8] POD 2025-10-01T07:31:56.591145237Z Gathering secrets in namespace openstack-operators [must-gather-mzhj8] POD 2025-10-01T07:31:57.205226887Z Gathering secrets in namespace baremetal-operator-system [must-gather-mzhj8] POD 2025-10-01T07:31:57.595099009Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:02.179381669Z Gathering secrets in namespace openshift-machine-api [must-gather-mzhj8] POD 2025-10-01T07:32:02.610909021Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:06.932387614Z Gathering csv [must-gather-mzhj8] POD 2025-10-01T07:32:07.078528967Z Gathering subscription [must-gather-mzhj8] POD 2025-10-01T07:32:07.231219658Z Gathering installplan [must-gather-mzhj8] POD 2025-10-01T07:32:07.642199652Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:07.714103538Z No resources found in openshift-machine-api namespace. [must-gather-mzhj8] POD 2025-10-01T07:32:07.772238551Z Dump logs for control-plane-machine-set-operator from control-plane-machine-set-operator-649bd778b4-tt5tw pod [must-gather-mzhj8] POD 2025-10-01T07:32:07.791453675Z Dump logs for kube-rbac-proxy from machine-api-operator-788b7c6b6c-ctdmb pod [must-gather-mzhj8] POD 2025-10-01T07:32:07.851918857Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather-mzhj8] POD 2025-10-01T07:32:07.890740774Z Dump logs for machine-api-operator from machine-api-operator-788b7c6b6c-ctdmb pod [must-gather-mzhj8] POD 2025-10-01T07:32:08.114279645Z Dump services: cluster-autoscaler-operator [must-gather-mzhj8] POD 2025-10-01T07:32:08.119523269Z Dump services: control-plane-machine-set-operator [must-gather-mzhj8] POD 2025-10-01T07:32:08.123112701Z Dump services: machine-api-controllers [must-gather-mzhj8] POD 2025-10-01T07:32:08.126478238Z Dump services: machine-api-operator [must-gather-mzhj8] POD 2025-10-01T07:32:08.134610717Z Dump services: machine-api-operator-machine-webhook [must-gather-mzhj8] POD 2025-10-01T07:32:08.272640131Z Dump services: machine-api-operator-webhook [must-gather-mzhj8] POD 2025-10-01T07:32:08.736540993Z Dump replicaset: control-plane-machine-set-operator-649bd778b4 [must-gather-mzhj8] POD 2025-10-01T07:32:08.743919523Z Dump replicaset: machine-api-operator-788b7c6b6c [must-gather-mzhj8] POD 2025-10-01T07:32:08.901635923Z Dump deployments: control-plane-machine-set-operator [must-gather-mzhj8] POD 2025-10-01T07:32:08.916038073Z Dump deployments: machine-api-operator [must-gather-mzhj8] POD 2025-10-01T07:32:09.731490313Z error: the server doesn't have a resource type "cert" [must-gather-mzhj8] POD 2025-10-01T07:32:09.933878780Z error: the server doesn't have a resource type "issuer" [must-gather-mzhj8] POD 2025-10-01T07:32:10.118961723Z error: the server doesn't have a resource type "certificaterequests" [must-gather-mzhj8] POD 2025-10-01T07:32:10.455420813Z Gathering secrets in namespace cert-manager [must-gather-mzhj8] POD 2025-10-01T07:32:11.144409976Z Gathering secrets in namespace openshift-nmstate [must-gather-mzhj8] POD 2025-10-01T07:32:11.827165329Z Gathering secrets in namespace openshift-operators-redhat [must-gather-mzhj8] POD 2025-10-01T07:32:12.512197810Z Gathering secrets in namespace openshift-logging [must-gather-mzhj8] POD 2025-10-01T07:32:12.661217696Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:13.319140892Z Gathering secrets in namespace metallb-system [must-gather-mzhj8] POD 2025-10-01T07:32:14.091653379Z error: the server doesn't have a resource type "nncp" [must-gather-mzhj8] POD 2025-10-01T07:32:14.277656256Z error: the server doesn't have a resource type "nnce" [must-gather-mzhj8] POD 2025-10-01T07:32:14.483361438Z error: the server doesn't have a resource type "nns" [must-gather-mzhj8] POD 2025-10-01T07:32:14.719942034Z error: the server doesn't have a resource type "ipaddresspools" [must-gather-mzhj8] POD 2025-10-01T07:32:15.411897253Z error: the server doesn't have a resource type "l2advertisement" [must-gather-mzhj8] POD 2025-10-01T07:32:15.424507957Z Gather ctlplane service info: openstack [must-gather-mzhj8] POD 2025-10-01T07:32:15.439258796Z Gather ctlplane service info: ovn [must-gather-mzhj8] POD 2025-10-01T07:32:15.593718312Z Error from server (NotFound): namespaces "openstack" not found [must-gather-mzhj8] POD 2025-10-01T07:32:15.597306064Z Error from server (NotFound): namespaces "openstack" not found [must-gather-mzhj8] POD 2025-10-01T07:32:15.607417004Z Pod ovsdbserver-nb-0 not found in namespace openstack, skipping OVN NB database collection [must-gather-mzhj8] POD 2025-10-01T07:32:15.786831420Z Pod ovsdbserver-sb-0 not found in namespace openstack, skipping OVN SB database collection [must-gather-mzhj8] POD 2025-10-01T07:32:15.793620564Z Gather ctlplane service info: rabbitmq [must-gather-mzhj8] POD 2025-10-01T07:32:15.942583760Z Error from server (NotFound): namespaces "openstack" not found [must-gather-mzhj8] POD 2025-10-01T07:32:16.125688702Z error: the server doesn't have a resource type "openstackcontrolplane" [must-gather-mzhj8] POD 2025-10-01T07:32:17.674511095Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:18.205451180Z tar: Removing leading `/' from member names [must-gather-mzhj8] POD 2025-10-01T07:32:22.691758485Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:27.708592174Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:32.728237256Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:37.746648732Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:42.762867634Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:47.777537077Z volume usage percentage 0 [must-gather-mzhj8] POD 2025-10-01T07:32:50.580675337Z The /must-gather/must-gather.tar.xz now can be attached to the support case. [must-gather-mzhj8] OUT 2025-10-01T07:32:51.590953579Z waiting for gather to complete [must-gather-mzhj8] OUT 2025-10-01T07:32:51.593924944Z downloading gather output [must-gather-mzhj8] OUT 2025-10-01T07:32:51.916597537Z receiving incremental file list [must-gather-mzhj8] OUT 2025-10-01T07:32:51.926267361Z ./ [must-gather-mzhj8] OUT 2025-10-01T07:32:51.927278137Z must-gather.tar.xz [must-gather-mzhj8] OUT 2025-10-01T07:32:51.995435387Z packagemanifests [must-gather-mzhj8] OUT 2025-10-01T07:32:52.715787255Z run.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716024551Z version [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716157305Z crd/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716193036Z csv/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716227177Z ctlplane/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716318769Z ctlplane/endpoints [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716458012Z ctlplane/services [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716536914Z ctlplane/ovn/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716593196Z ctlplane/rabbitmq/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716628837Z dbs/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716664208Z namespaces/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716737449Z namespaces/openshift-machine-api/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.716916434Z namespaces/openshift-machine-api/all_resources.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717103989Z namespaces/openshift-machine-api/events.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717379416Z namespaces/openshift-machine-api/nad.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.71753846Z namespaces/openshift-machine-api/pvc.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717637432Z namespaces/openshift-machine-api/buildconfig/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717674993Z namespaces/openshift-machine-api/cert/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717708074Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717752985Z namespaces/openshift-machine-api/configmaps/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717793946Z namespaces/openshift-machine-api/cronjobs/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717818657Z namespaces/openshift-machine-api/daemonset/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717882048Z namespaces/openshift-machine-api/deployments/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.717995781Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718172806Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718323349Z namespaces/openshift-machine-api/imagestream/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.71835733Z namespaces/openshift-machine-api/installplans/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718389931Z namespaces/openshift-machine-api/issuer/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718432302Z namespaces/openshift-machine-api/jobs/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718473193Z namespaces/openshift-machine-api/pods/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718501994Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718632547Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718796951Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.718931385Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719103879Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719223712Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719389426Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719491689Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719665423Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.719808367Z namespaces/openshift-machine-api/replicaset/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.71992988Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720090914Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720214437Z namespaces/openshift-machine-api/routes/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720254278Z namespaces/openshift-machine-api/services/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720356501Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720532685Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72072816Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.720918675Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721156201Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721330795Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721448078Z namespaces/openshift-machine-api/statefulset/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721493459Z namespaces/openshift-machine-api/subscriptions/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72152084Z network/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721617863Z network/l2advertisement.log [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721691564Z network/ipaddresspools/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721739106Z network/nnce/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721768816Z network/nncp/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.721799077Z network/nns/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.7223297Z nodes/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.722480654Z nodes/crc.describe [must-gather-mzhj8] OUT 2025-10-01T07:32:52.722742541Z nodes/crc.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.722979727Z sos-reports/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.723041598Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72308311Z sos-reports/_all_nodes/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72309172Z webhooks/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72309943Z webhooks/mutating/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72310695Z webhooks/validating/ [must-gather-mzhj8] OUT 2025-10-01T07:32:52.723124791Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.723281865Z webhooks/validating/multus.openshift.io.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.723443059Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-mzhj8] OUT 2025-10-01T07:32:52.72548735Z [must-gather-mzhj8] OUT 2025-10-01T07:32:52.725533601Z sent 787 bytes received 24,994,632 bytes 16,663,612.67 bytes/sec [must-gather-mzhj8] OUT 2025-10-01T07:32:52.725542022Z total size is 174,366,845 speedup is 6.98 [must-gather ] OUT 2025-10-01T07:32:52.753355584Z namespace/openshift-must-gather-6nk2b deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43221", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003a9d398), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because WellKnownReadyControllerDegraded: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43221", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003a9d398), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly) clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing