[must-gather ] OUT 2025-11-26T07:09:17.56081731Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42557", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc001a02a08), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/network is degraded because Error while updating operator configuration: could not apply (apps/v1, Kind=Deployment) openshift-network-diagnostics/network-check-source: failed to apply / update (apps/v1, Kind=Deployment) openshift-network-diagnostics/network-check-source: Patch "https://api-int.crc.testing:6443/apis/apps/v1/namespaces/openshift-network-diagnostics/deployments/network-check-source?fieldManager=cluster-network-operator%2Foperconfig&force=true": dial tcp 38.102.83.136:6443: connect: connection refused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-26T07:09:17.690250922Z namespace/openshift-must-gather-qqtxm created [must-gather ] OUT 2025-11-26T07:09:17.69354642Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-7mdt8 created [must-gather ] OUT 2025-11-26T07:09:17.804985658Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-kpl6r] OUT 2025-11-26T07:11:16.94371376Z waiting for gather to complete [must-gather-kpl6r] OUT 2025-11-26T07:11:16.946651459Z downloading gather output [must-gather-kpl6r] OUT 2025-11-26T07:11:17.252493247Z receiving incremental file list [must-gather-kpl6r] OUT 2025-11-26T07:11:17.261182122Z ./ [must-gather-kpl6r] OUT 2025-11-26T07:11:17.261371148Z must-gather.tar.xz [must-gather-kpl6r] OUT 2025-11-26T07:11:17.326551659Z packagemanifests [must-gather-kpl6r] OUT 2025-11-26T07:11:18.027262384Z run.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.027571353Z version [must-gather-kpl6r] OUT 2025-11-26T07:11:18.027781498Z crd/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.027824979Z csv/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.02786131Z ctlplane/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.02823477Z ctlplane/endpoints [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028408915Z ctlplane/services [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028491938Z ctlplane/ovn/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028550889Z ctlplane/rabbitmq/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.0285776Z dbs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028606531Z namespaces/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028631301Z namespaces/openshift-machine-api/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.028906549Z namespaces/openshift-machine-api/all_resources.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.029097004Z namespaces/openshift-machine-api/events.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.029685709Z namespaces/openshift-machine-api/nad.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.029898996Z namespaces/openshift-machine-api/pvc.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.029991338Z namespaces/openshift-machine-api/buildconfig/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030022669Z namespaces/openshift-machine-api/cert/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.03007248Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030094081Z namespaces/openshift-machine-api/configmaps/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030118392Z namespaces/openshift-machine-api/cronjobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030155222Z namespaces/openshift-machine-api/daemonset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030174303Z namespaces/openshift-machine-api/deployments/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030267315Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030796769Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030935043Z namespaces/openshift-machine-api/imagestream/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030961674Z namespaces/openshift-machine-api/installplans/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.030994015Z namespaces/openshift-machine-api/issuer/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031020225Z namespaces/openshift-machine-api/jobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031097047Z namespaces/openshift-machine-api/pods/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031122418Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031264223Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031441567Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.031559971Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032068004Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032168047Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-kpl6r] OUT 2025-11-26T07:11:18.03230648Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032402123Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032583407Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032713312Z namespaces/openshift-machine-api/replicaset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032802514Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.032952248Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.033575035Z namespaces/openshift-machine-api/routes/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.033618706Z namespaces/openshift-machine-api/services/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.033747109Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.033995566Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.037676766Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.037802039Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038026425Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038265382Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038371355Z namespaces/openshift-machine-api/statefulset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038451257Z namespaces/openshift-machine-api/subscriptions/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038538189Z namespaces/openstack-operators/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038635252Z namespaces/openstack-operators/all_resources.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038729944Z namespaces/openstack-operators/events.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.038829027Z namespaces/openstack-operators/nad.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.03893171Z namespaces/openstack-operators/pvc.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039022602Z namespaces/openstack-operators/buildconfig/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039135885Z namespaces/openstack-operators/cert/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039234428Z namespaces/openstack-operators/certificaterequests/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.03933506Z namespaces/openstack-operators/configmaps/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039444173Z namespaces/openstack-operators/cronjobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039546056Z namespaces/openstack-operators/daemonset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039629148Z namespaces/openstack-operators/deployments/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039714491Z namespaces/openstack-operators/imagestream/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039798523Z namespaces/openstack-operators/installplans/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039910876Z namespaces/openstack-operators/issuer/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.039993878Z namespaces/openstack-operators/jobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040099651Z namespaces/openstack-operators/pods/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040276246Z namespaces/openstack-operators/pods/-describe [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040289356Z namespaces/openstack-operators/pods/logs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040293976Z namespaces/openstack-operators/replicaset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040297926Z namespaces/openstack-operators/routes/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040303397Z namespaces/openstack-operators/services/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040307167Z namespaces/openstack-operators/statefulset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040310987Z namespaces/openstack-operators/subscriptions/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040315457Z namespaces/openstack/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040331527Z namespaces/openstack/all_resources.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040335467Z namespaces/openstack/events.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040339388Z namespaces/openstack/nad.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040343458Z namespaces/openstack/pvc.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040347438Z namespaces/openstack/buildconfig/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040351268Z namespaces/openstack/cert/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040355698Z namespaces/openstack/certificaterequests/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040367128Z namespaces/openstack/configmaps/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040371778Z namespaces/openstack/cronjobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040376509Z namespaces/openstack/daemonset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040380689Z namespaces/openstack/deployments/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040384579Z namespaces/openstack/imagestream/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040388689Z namespaces/openstack/installplans/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040392969Z namespaces/openstack/issuer/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040397279Z namespaces/openstack/jobs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040401689Z namespaces/openstack/pods/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040406049Z namespaces/openstack/pods/-describe [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040410289Z namespaces/openstack/pods/logs/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.04041412Z namespaces/openstack/replicaset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.04041796Z namespaces/openstack/routes/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.0404218Z namespaces/openstack/services/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.04042555Z namespaces/openstack/statefulset/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.04043012Z namespaces/openstack/subscriptions/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.04043512Z network/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040473241Z network/l2advertisement.log [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040538463Z network/ipaddresspools/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040583494Z network/nnce/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040604285Z network/nncp/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040621825Z network/nns/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.040641326Z nodes/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041280763Z nodes/crc.describe [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041570761Z nodes/crc.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041703574Z sos-reports/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041734585Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041752016Z sos-reports/_all_nodes/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041796547Z webhooks/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041816727Z webhooks/mutating/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.041833128Z webhooks/validating/ [must-gather-kpl6r] OUT 2025-11-26T07:11:18.0419072Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.042079644Z webhooks/validating/multus.openshift.io.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.042311591Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-kpl6r] OUT 2025-11-26T07:11:18.045635181Z [must-gather-kpl6r] OUT 2025-11-26T07:11:18.045697903Z sent 1,129 bytes received 26,677,147 bytes 10,671,310.40 bytes/sec [must-gather-kpl6r] OUT 2025-11-26T07:11:18.045721963Z total size is 184,036,919 speedup is 6.90 [must-gather ] OUT 2025-11-26T07:11:18.07297533Z namespace/openshift-must-gather-qqtxm deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing