[must-gather ] OUT 2025-10-12T00:28:16.936645424Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43193", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00378fc20), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-12T00:28:17.008590786Z namespace/openshift-must-gather-2b9t8 created [must-gather ] OUT 2025-10-12T00:28:17.019586814Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-747ws created [must-gather ] OUT 2025-10-12T00:28:17.034379378Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-87glr] OUT 2025-10-12T00:29:58.570800355Z waiting for gather to complete [must-gather-87glr] OUT 2025-10-12T00:29:58.57388473Z downloading gather output [must-gather-87glr] OUT 2025-10-12T00:29:58.895601397Z receiving incremental file list [must-gather-87glr] OUT 2025-10-12T00:29:58.90485974Z ./ [must-gather-87glr] OUT 2025-10-12T00:29:58.905122537Z must-gather.tar.xz [must-gather-87glr] OUT 2025-10-12T00:29:58.962383295Z packagemanifests [must-gather-87glr] OUT 2025-10-12T00:29:59.752591428Z run.log [must-gather-87glr] OUT 2025-10-12T00:29:59.75266698Z version [must-gather-87glr] OUT 2025-10-12T00:29:59.752706961Z crd/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752712471Z csv/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752722472Z ctlplane/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752732122Z ctlplane/endpoints [must-gather-87glr] OUT 2025-10-12T00:29:59.752738672Z ctlplane/services [must-gather-87glr] OUT 2025-10-12T00:29:59.752748392Z ctlplane/ovn/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752755863Z ctlplane/rabbitmq/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752760963Z dbs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752766333Z namespaces/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752771553Z namespaces/openshift-machine-api/ [must-gather-87glr] OUT 2025-10-12T00:29:59.752831085Z namespaces/openshift-machine-api/all_resources.log [must-gather-87glr] OUT 2025-10-12T00:29:59.752952179Z namespaces/openshift-machine-api/events.log [must-gather-87glr] OUT 2025-10-12T00:29:59.753175536Z namespaces/openshift-machine-api/nad.log [must-gather-87glr] OUT 2025-10-12T00:29:59.753226296Z namespaces/openshift-machine-api/pvc.log [must-gather-87glr] OUT 2025-10-12T00:29:59.753256437Z namespaces/openshift-machine-api/buildconfig/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753261337Z namespaces/openshift-machine-api/cert/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753266227Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753270298Z namespaces/openshift-machine-api/configmaps/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753315229Z namespaces/openshift-machine-api/cronjobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75335405Z namespaces/openshift-machine-api/daemonset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75336258Z namespaces/openshift-machine-api/deployments/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753377821Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.753439793Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.753485534Z namespaces/openshift-machine-api/imagestream/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753492684Z namespaces/openshift-machine-api/installplans/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753497925Z namespaces/openshift-machine-api/issuer/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753503445Z namespaces/openshift-machine-api/jobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753507975Z namespaces/openshift-machine-api/pods/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753522515Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753585097Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-87glr] OUT 2025-10-12T00:29:59.753647589Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753707471Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-87glr] OUT 2025-10-12T00:29:59.753802804Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753832695Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-87glr] OUT 2025-10-12T00:29:59.753922798Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.753940588Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-87glr] OUT 2025-10-12T00:29:59.754083493Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-87glr] OUT 2025-10-12T00:29:59.754159975Z namespaces/openshift-machine-api/replicaset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.754185526Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754285959Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.75432477Z namespaces/openshift-machine-api/routes/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75433227Z namespaces/openshift-machine-api/services/ [must-gather-87glr] OUT 2025-10-12T00:29:59.754379062Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754466815Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754561587Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754620229Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754776094Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754882837Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.754909968Z namespaces/openshift-machine-api/statefulset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.754919239Z namespaces/openshift-machine-api/subscriptions/ [must-gather-87glr] OUT 2025-10-12T00:29:59.754923219Z namespaces/openstack-operators/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75496051Z namespaces/openstack-operators/all_resources.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755038342Z namespaces/openstack-operators/events.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755154676Z namespaces/openstack-operators/nad.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755233678Z namespaces/openstack-operators/pvc.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755263409Z namespaces/openstack-operators/buildconfig/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75527566Z namespaces/openstack-operators/cert/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75528304Z namespaces/openstack-operators/certificaterequests/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75528955Z namespaces/openstack-operators/configmaps/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75530135Z namespaces/openstack-operators/cronjobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755311271Z namespaces/openstack-operators/daemonset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755317301Z namespaces/openstack-operators/deployments/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755323991Z namespaces/openstack-operators/imagestream/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755330471Z namespaces/openstack-operators/installplans/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755339491Z namespaces/openstack-operators/issuer/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755348022Z namespaces/openstack-operators/jobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755355482Z namespaces/openstack-operators/pods/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755391713Z namespaces/openstack-operators/pods/-describe [must-gather-87glr] OUT 2025-10-12T00:29:59.755404113Z namespaces/openstack-operators/pods/logs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755411534Z namespaces/openstack-operators/replicaset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755418424Z namespaces/openstack-operators/routes/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755425674Z namespaces/openstack-operators/services/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755431814Z namespaces/openstack-operators/statefulset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755438905Z namespaces/openstack-operators/subscriptions/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755446005Z namespaces/openstack/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755471486Z namespaces/openstack/all_resources.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755499696Z namespaces/openstack/events.log [must-gather-87glr] OUT 2025-10-12T00:29:59.75560956Z namespaces/openstack/nad.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755633301Z namespaces/openstack/pvc.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755643071Z namespaces/openstack/buildconfig/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755648801Z namespaces/openstack/cert/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755653111Z namespaces/openstack/certificaterequests/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755668742Z namespaces/openstack/configmaps/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755681362Z namespaces/openstack/cronjobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755687282Z namespaces/openstack/daemonset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755694932Z namespaces/openstack/deployments/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755702163Z namespaces/openstack/imagestream/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755709533Z namespaces/openstack/installplans/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755716363Z namespaces/openstack/issuer/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755721953Z namespaces/openstack/jobs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755739684Z namespaces/openstack/pods/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755754804Z namespaces/openstack/pods/-describe [must-gather-87glr] OUT 2025-10-12T00:29:59.755765615Z namespaces/openstack/pods/logs/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755772535Z namespaces/openstack/replicaset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755779495Z namespaces/openstack/routes/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755785245Z namespaces/openstack/services/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755791095Z namespaces/openstack/statefulset/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755798806Z namespaces/openstack/subscriptions/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755806356Z network/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755828977Z network/l2advertisement.log [must-gather-87glr] OUT 2025-10-12T00:29:59.755835777Z network/ipaddresspools/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755842147Z network/nnce/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755847807Z network/nncp/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755854877Z network/nns/ [must-gather-87glr] OUT 2025-10-12T00:29:59.755862528Z nodes/ [must-gather-87glr] OUT 2025-10-12T00:29:59.75593999Z nodes/crc.describe [must-gather-87glr] OUT 2025-10-12T00:29:59.756042172Z nodes/crc.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.756152656Z sos-reports/ [must-gather-87glr] OUT 2025-10-12T00:29:59.756164196Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-87glr] OUT 2025-10-12T00:29:59.756169886Z sos-reports/_all_nodes/ [must-gather-87glr] OUT 2025-10-12T00:29:59.756181076Z webhooks/ [must-gather-87glr] OUT 2025-10-12T00:29:59.756185487Z webhooks/mutating/ [must-gather-87glr] OUT 2025-10-12T00:29:59.756189577Z webhooks/validating/ [must-gather-87glr] OUT 2025-10-12T00:29:59.756193987Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.75629049Z webhooks/validating/multus.openshift.io.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.756365552Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-87glr] OUT 2025-10-12T00:29:59.76054223Z [must-gather-87glr] OUT 2025-10-12T00:29:59.760610432Z sent 1,129 bytes received 26,809,305 bytes 17,873,622.67 bytes/sec [must-gather-87glr] OUT 2025-10-12T00:29:59.760615412Z total size is 178,114,669 speedup is 6.64 [must-gather ] OUT 2025-10-12T00:29:59.789339119Z namespace/openshift-must-gather-2b9t8 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing