[must-gather ] OUT 2025-12-29T00:16:51.204553982Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.8 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42813", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003e0d278), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-29T00:16:51.260091131Z namespace/openshift-must-gather-22gr9 created [must-gather ] OUT 2025-12-29T00:16:51.265835109Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-zgt6r created [must-gather ] OUT 2025-12-29T00:16:51.474918755Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-6vhlm] OUT 2025-12-29T00:18:26.74867805Z waiting for gather to complete [must-gather-6vhlm] OUT 2025-12-29T00:18:26.751372134Z downloading gather output [must-gather-6vhlm] OUT 2025-12-29T00:18:27.033095827Z receiving incremental file list [must-gather-6vhlm] OUT 2025-12-29T00:18:27.044171071Z ./ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.044304974Z must-gather.tar.xz [must-gather-6vhlm] OUT 2025-12-29T00:18:27.090509445Z packagemanifests [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864811463Z run.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864891645Z version [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864907766Z crd/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864919706Z csv/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864931806Z ctlplane/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.864956227Z ctlplane/endpoints [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865035138Z ctlplane/services [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865056109Z ctlplane/ovn/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865065579Z ctlplane/rabbitmq/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865074759Z dbs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86508473Z namespaces/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86509417Z namespaces/openshift-machine-api/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865108841Z namespaces/openshift-machine-api/all_resources.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865304446Z namespaces/openshift-machine-api/events.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865525901Z namespaces/openshift-machine-api/nad.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865639973Z namespaces/openshift-machine-api/pvc.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865662054Z namespaces/openshift-machine-api/buildconfig/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865693135Z namespaces/openshift-machine-api/cert/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865705215Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865716835Z namespaces/openshift-machine-api/configmaps/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865728115Z namespaces/openshift-machine-api/cronjobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865739536Z namespaces/openshift-machine-api/daemonset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865750826Z namespaces/openshift-machine-api/deployments/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865769296Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865882009Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86593314Z namespaces/openshift-machine-api/imagestream/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865945391Z namespaces/openshift-machine-api/installplans/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865961491Z namespaces/openshift-machine-api/issuer/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.865972531Z namespaces/openshift-machine-api/jobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866032493Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866043893Z namespaces/openshift-machine-api/pods/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866055363Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866068803Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866122955Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866185586Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866318149Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86633821Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866509915Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866539025Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866712519Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866811852Z namespaces/openshift-machine-api/replicaset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.866862183Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867016116Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867066707Z namespaces/openshift-machine-api/routes/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867079008Z namespaces/openshift-machine-api/services/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867137349Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867295743Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867389455Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867512888Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867639181Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867735313Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867754513Z namespaces/openshift-machine-api/statefulset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867765234Z namespaces/openshift-machine-api/subscriptions/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867783294Z namespaces/openstack-operators/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867799934Z namespaces/openstack-operators/all_resources.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867877516Z namespaces/openstack-operators/events.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.867931558Z namespaces/openstack-operators/nad.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86801584Z namespaces/openstack-operators/pvc.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868032931Z namespaces/openstack-operators/buildconfig/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868043661Z namespaces/openstack-operators/cert/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868059631Z namespaces/openstack-operators/certificaterequests/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868072892Z namespaces/openstack-operators/configmaps/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868083592Z namespaces/openstack-operators/cronjobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868094002Z namespaces/openstack-operators/daemonset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868104422Z namespaces/openstack-operators/deployments/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868114663Z namespaces/openstack-operators/imagestream/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868125033Z namespaces/openstack-operators/installplans/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868135233Z namespaces/openstack-operators/issuer/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868145493Z namespaces/openstack-operators/jobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868155654Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868165784Z namespaces/openstack-operators/pods/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868182454Z namespaces/openstack-operators/pods/-describe [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868192934Z namespaces/openstack-operators/pods/logs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868203245Z namespaces/openstack-operators/replicaset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868213415Z namespaces/openstack-operators/routes/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868223615Z namespaces/openstack-operators/services/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868233815Z namespaces/openstack-operators/statefulset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868243876Z namespaces/openstack-operators/subscriptions/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868253856Z namespaces/openstack/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868264096Z namespaces/openstack/all_resources.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868280027Z namespaces/openstack/events.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868297747Z namespaces/openstack/nad.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868400349Z namespaces/openstack/pvc.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86841642Z namespaces/openstack/buildconfig/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86842587Z namespaces/openstack/cert/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.8684408Z namespaces/openstack/certificaterequests/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.86845151Z namespaces/openstack/configmaps/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868462641Z namespaces/openstack/cronjobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868551803Z namespaces/openstack/daemonset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868581963Z namespaces/openstack/deployments/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868592834Z namespaces/openstack/imagestream/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868605934Z namespaces/openstack/installplans/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868627775Z namespaces/openstack/issuer/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868638195Z namespaces/openstack/jobs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868650125Z namespaces/openstack/poddisruptionbudgets/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868660175Z namespaces/openstack/pods/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868684376Z namespaces/openstack/pods/-describe [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868695716Z namespaces/openstack/pods/logs/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868706346Z namespaces/openstack/replicaset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868716877Z namespaces/openstack/routes/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868727237Z namespaces/openstack/services/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868737597Z namespaces/openstack/statefulset/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868747777Z namespaces/openstack/subscriptions/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868758008Z network/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868786628Z network/l2advertisement.log [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868797438Z network/ipaddresspools/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868808359Z network/nnce/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868818829Z network/nncp/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868828909Z network/nns/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868839399Z nodes/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.8688497Z nodes/crc.describe [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868890081Z nodes/crc.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868907841Z sos-reports/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868918251Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868928431Z sos-reports/_all_nodes/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868940042Z webhooks/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868950342Z webhooks/mutating/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868960802Z webhooks/validating/ [must-gather-6vhlm] OUT 2025-12-29T00:18:27.868977763Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.869033504Z webhooks/validating/multus.openshift.io.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.869175937Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-6vhlm] OUT 2025-12-29T00:18:27.872940007Z [must-gather-6vhlm] OUT 2025-12-29T00:18:27.872955448Z sent 1,141 bytes received 26,536,253 bytes 17,691,596.00 bytes/sec [must-gather-6vhlm] OUT 2025-12-29T00:18:27.872960798Z total size is 187,320,981 speedup is 7.06 [must-gather ] OUT 2025-12-29T00:18:27.900808531Z namespace/openshift-must-gather-22gr9 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.8 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/controller-manager: observed generation is 19, desired generation is 20. Progressing: deployment/route-controller-manager: observed generation is 17, desired generation is 18. clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing