[must-gather ] OUT 2025-12-02T00:17:11.559004143Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42942", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0015fd8a8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-02T00:17:11.607731463Z namespace/openshift-must-gather-jfbsl created [must-gather ] OUT 2025-12-02T00:17:11.611757837Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-bghk2 created [must-gather ] OUT 2025-12-02T00:17:11.806488708Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-j5d84] OUT 2025-12-02T00:18:43.432658122Z waiting for gather to complete [must-gather-j5d84] OUT 2025-12-02T00:18:43.435455442Z downloading gather output [must-gather-j5d84] OUT 2025-12-02T00:18:43.672756669Z receiving incremental file list [must-gather-j5d84] OUT 2025-12-02T00:18:43.681934069Z ./ [must-gather-j5d84] OUT 2025-12-02T00:18:43.682268799Z must-gather.tar.xz [must-gather-j5d84] OUT 2025-12-02T00:18:43.765184476Z packagemanifests [must-gather-j5d84] OUT 2025-12-02T00:18:44.456740804Z run.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.456833636Z version [must-gather-j5d84] OUT 2025-12-02T00:18:44.456888678Z crd/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.456896548Z csv/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.456900438Z ctlplane/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.45696664Z ctlplane/endpoints [must-gather-j5d84] OUT 2025-12-02T00:18:44.457074183Z ctlplane/services [must-gather-j5d84] OUT 2025-12-02T00:18:44.457169806Z ctlplane/ovn/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457178466Z ctlplane/rabbitmq/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457183316Z dbs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457188886Z namespaces/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457193917Z namespaces/openshift-machine-api/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457213127Z namespaces/openshift-machine-api/all_resources.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.457423793Z namespaces/openshift-machine-api/events.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.457605588Z namespaces/openshift-machine-api/nad.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.457731562Z namespaces/openshift-machine-api/pvc.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.457762403Z namespaces/openshift-machine-api/buildconfig/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457771233Z namespaces/openshift-machine-api/cert/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457776783Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457781503Z namespaces/openshift-machine-api/configmaps/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457788563Z namespaces/openshift-machine-api/cronjobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457792864Z namespaces/openshift-machine-api/daemonset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457797934Z namespaces/openshift-machine-api/deployments/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.457847825Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.45801031Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.458105612Z namespaces/openshift-machine-api/imagestream/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458118513Z namespaces/openshift-machine-api/installplans/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458122653Z namespaces/openshift-machine-api/issuer/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458127053Z namespaces/openshift-machine-api/jobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458130893Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458135093Z namespaces/openshift-machine-api/pods/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458142103Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458165224Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-j5d84] OUT 2025-12-02T00:18:44.458326689Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.45835725Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.458481053Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.458529784Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-j5d84] OUT 2025-12-02T00:18:44.458656488Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.45873767Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.458875574Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.458951086Z namespaces/openshift-machine-api/replicaset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.459001708Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.459147152Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.459226744Z namespaces/openshift-machine-api/routes/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.459264075Z namespaces/openshift-machine-api/services/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.459307236Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.45942153Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.459645846Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.459661766Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.460045727Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.460103909Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.460108609Z namespaces/openshift-machine-api/statefulset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460118989Z namespaces/openshift-machine-api/subscriptions/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46012656Z namespaces/openstack-operators/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46013068Z namespaces/openstack-operators/all_resources.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460211492Z namespaces/openstack-operators/events.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460319935Z namespaces/openstack-operators/nad.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460431118Z namespaces/openstack-operators/pvc.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460475089Z namespaces/openstack-operators/buildconfig/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46048527Z namespaces/openstack-operators/cert/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46050177Z namespaces/openstack-operators/certificaterequests/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460512831Z namespaces/openstack-operators/configmaps/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460517751Z namespaces/openstack-operators/cronjobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460523481Z namespaces/openstack-operators/daemonset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460527391Z namespaces/openstack-operators/deployments/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460531221Z namespaces/openstack-operators/imagestream/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460535121Z namespaces/openstack-operators/installplans/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460538821Z namespaces/openstack-operators/issuer/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460543521Z namespaces/openstack-operators/jobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460548332Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460553252Z namespaces/openstack-operators/pods/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460561172Z namespaces/openstack-operators/pods/-describe [must-gather-j5d84] OUT 2025-12-02T00:18:44.460644814Z namespaces/openstack-operators/pods/logs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460658525Z namespaces/openstack-operators/replicaset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460663635Z namespaces/openstack-operators/routes/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460668885Z namespaces/openstack-operators/services/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460673315Z namespaces/openstack-operators/statefulset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460677095Z namespaces/openstack-operators/subscriptions/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460681135Z namespaces/openstack/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.460688786Z namespaces/openstack/all_resources.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460770458Z namespaces/openstack/events.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.46086024Z namespaces/openstack/nad.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.460971634Z namespaces/openstack/pvc.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.461013615Z namespaces/openstack/buildconfig/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461021965Z namespaces/openstack/cert/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461026845Z namespaces/openstack/certificaterequests/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461035605Z namespaces/openstack/configmaps/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461040715Z namespaces/openstack/cronjobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461045556Z namespaces/openstack/daemonset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461049516Z namespaces/openstack/deployments/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461053736Z namespaces/openstack/imagestream/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461069646Z namespaces/openstack/installplans/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461088707Z namespaces/openstack/issuer/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461092947Z namespaces/openstack/jobs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461096677Z namespaces/openstack/poddisruptionbudgets/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461103207Z namespaces/openstack/pods/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461113528Z namespaces/openstack/pods/-describe [must-gather-j5d84] OUT 2025-12-02T00:18:44.461163479Z namespaces/openstack/pods/logs/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461169999Z namespaces/openstack/replicaset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461174299Z namespaces/openstack/routes/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461179369Z namespaces/openstack/services/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46118771Z namespaces/openstack/statefulset/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.4611922Z namespaces/openstack/subscriptions/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.46119725Z network/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461222671Z network/l2advertisement.log [must-gather-j5d84] OUT 2025-12-02T00:18:44.461271242Z network/ipaddresspools/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461319023Z network/nnce/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461331454Z network/nncp/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461351034Z network/nns/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461356024Z nodes/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461360755Z nodes/crc.describe [must-gather-j5d84] OUT 2025-12-02T00:18:44.461580041Z nodes/crc.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.461704424Z sos-reports/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461718365Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-j5d84] OUT 2025-12-02T00:18:44.461723825Z sos-reports/_all_nodes/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461738445Z webhooks/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461742665Z webhooks/mutating/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461749326Z webhooks/validating/ [must-gather-j5d84] OUT 2025-12-02T00:18:44.461800607Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.461925621Z webhooks/validating/multus.openshift.io.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.462042414Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-j5d84] OUT 2025-12-02T00:18:44.46439585Z [must-gather-j5d84] OUT 2025-12-02T00:18:44.464417001Z sent 1,145 bytes received 27,025,819 bytes 18,017,976.00 bytes/sec [must-gather-j5d84] OUT 2025-12-02T00:18:44.464421951Z total size is 184,793,952 speedup is 6.84 [must-gather ] OUT 2025-12-02T00:18:44.49369394Z namespace/openshift-must-gather-jfbsl deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/controller-manager: observed generation is 18, desired generation is 19. clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing