[must-gather ] OUT 2025-12-29T00:24:20.136743524Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.8 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42862", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc003c915f0), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-12-29T00:24:20.186769571Z namespace/openshift-must-gather-qczlj created [must-gather ] OUT 2025-12-29T00:24:20.190400513Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-97882 created [must-gather ] OUT 2025-12-29T00:24:20.381821463Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-8jssz] OUT 2025-12-29T00:26:11.754376168Z waiting for gather to complete [must-gather-8jssz] OUT 2025-12-29T00:26:11.757594097Z downloading gather output [must-gather-8jssz] OUT 2025-12-29T00:26:12.027349024Z receiving incremental file list [must-gather-8jssz] OUT 2025-12-29T00:26:12.038352634Z ./ [must-gather-8jssz] OUT 2025-12-29T00:26:12.03855661Z must-gather.tar.xz [must-gather-8jssz] OUT 2025-12-29T00:26:12.102475595Z packagemanifests [must-gather-8jssz] OUT 2025-12-29T00:26:12.906021826Z run.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.906880273Z version [must-gather-8jssz] OUT 2025-12-29T00:26:12.906900434Z crd/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906908174Z csv/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906915984Z ctlplane/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906928795Z ctlplane/endpoints [must-gather-8jssz] OUT 2025-12-29T00:26:12.906940605Z ctlplane/services [must-gather-8jssz] OUT 2025-12-29T00:26:12.906947425Z ctlplane/ovn/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906956815Z ctlplane/rabbitmq/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906963796Z dbs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906970456Z namespaces/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906977716Z namespaces/openshift-machine-api/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.906984566Z namespaces/openshift-machine-api/all_resources.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.906991727Z namespaces/openshift-machine-api/events.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.90710645Z namespaces/openshift-machine-api/nad.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.907162871Z namespaces/openshift-machine-api/pvc.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.907252964Z namespaces/openshift-machine-api/buildconfig/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907264324Z namespaces/openshift-machine-api/cert/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907271984Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907278225Z namespaces/openshift-machine-api/configmaps/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907288355Z namespaces/openshift-machine-api/cronjobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907296125Z namespaces/openshift-machine-api/daemonset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907302465Z namespaces/openshift-machine-api/deployments/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.907343137Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.90745814Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.908057589Z namespaces/openshift-machine-api/imagestream/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.90807656Z namespaces/openshift-machine-api/installplans/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.90808792Z namespaces/openshift-machine-api/issuer/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.90809892Z namespaces/openshift-machine-api/jobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908110061Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908121031Z namespaces/openshift-machine-api/pods/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908163742Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908473502Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-8jssz] OUT 2025-12-29T00:26:12.908488372Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908500923Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.908513643Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908524464Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-8jssz] OUT 2025-12-29T00:26:12.908535574Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908574365Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.908622977Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.908636857Z namespaces/openshift-machine-api/replicaset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908647397Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.908658418Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.908686079Z namespaces/openshift-machine-api/routes/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.908697469Z namespaces/openshift-machine-api/services/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.90872306Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.909021489Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.90905009Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.90906288Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.909295898Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.909319698Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.90935477Z namespaces/openshift-machine-api/statefulset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.90937185Z namespaces/openshift-machine-api/subscriptions/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.9093788Z namespaces/openstack-operators/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909390601Z namespaces/openstack-operators/all_resources.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.909557646Z namespaces/openstack-operators/events.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.909577756Z namespaces/openstack-operators/nad.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.909717291Z namespaces/openstack-operators/pvc.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.909734691Z namespaces/openstack-operators/buildconfig/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909748092Z namespaces/openstack-operators/cert/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909758912Z namespaces/openstack-operators/certificaterequests/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909770282Z namespaces/openstack-operators/configmaps/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909781063Z namespaces/openstack-operators/cronjobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909791113Z namespaces/openstack-operators/daemonset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909801803Z namespaces/openstack-operators/deployments/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909812234Z namespaces/openstack-operators/imagestream/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909822874Z namespaces/openstack-operators/installplans/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909832964Z namespaces/openstack-operators/issuer/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909843435Z namespaces/openstack-operators/jobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909853675Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909863945Z namespaces/openstack-operators/pods/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909892116Z namespaces/openstack-operators/pods/-describe [must-gather-8jssz] OUT 2025-12-29T00:26:12.909931577Z namespaces/openstack-operators/pods/logs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909944017Z namespaces/openstack-operators/replicaset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909954147Z namespaces/openstack-operators/routes/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909964228Z namespaces/openstack-operators/services/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909974258Z namespaces/openstack-operators/statefulset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909987528Z namespaces/openstack-operators/subscriptions/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.909998319Z namespaces/openstack/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910008689Z namespaces/openstack/all_resources.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.91003734Z namespaces/openstack/events.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.91005428Z namespaces/openstack/nad.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.910515875Z namespaces/openstack/pvc.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.910544226Z namespaces/openstack/buildconfig/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910556706Z namespaces/openstack/cert/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910569856Z namespaces/openstack/certificaterequests/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910581127Z namespaces/openstack/configmaps/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910599437Z namespaces/openstack/cronjobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910608288Z namespaces/openstack/daemonset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910616878Z namespaces/openstack/deployments/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910626818Z namespaces/openstack/imagestream/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910637409Z namespaces/openstack/installplans/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910661299Z namespaces/openstack/issuer/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.91067217Z namespaces/openstack/jobs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.91068242Z namespaces/openstack/poddisruptionbudgets/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.91069321Z namespaces/openstack/pods/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910703921Z namespaces/openstack/pods/-describe [must-gather-8jssz] OUT 2025-12-29T00:26:12.910714281Z namespaces/openstack/pods/logs/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910724521Z namespaces/openstack/replicaset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910733722Z namespaces/openstack/routes/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910742162Z namespaces/openstack/services/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910754242Z namespaces/openstack/statefulset/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910765453Z namespaces/openstack/subscriptions/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910778203Z network/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910790343Z network/l2advertisement.log [must-gather-8jssz] OUT 2025-12-29T00:26:12.910802214Z network/ipaddresspools/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910812864Z network/nnce/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910823274Z network/nncp/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910833965Z network/nns/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910844155Z nodes/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910852605Z nodes/crc.describe [must-gather-8jssz] OUT 2025-12-29T00:26:12.910886616Z nodes/crc.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.910904257Z sos-reports/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910914867Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-8jssz] OUT 2025-12-29T00:26:12.910925768Z sos-reports/_all_nodes/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910936718Z webhooks/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910947118Z webhooks/mutating/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910957149Z webhooks/validating/ [must-gather-8jssz] OUT 2025-12-29T00:26:12.910967339Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.910977399Z webhooks/validating/multus.openshift.io.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.91100462Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-8jssz] OUT 2025-12-29T00:26:12.915641763Z [must-gather-8jssz] OUT 2025-12-29T00:26:12.915673474Z sent 1,137 bytes received 27,024,937 bytes 18,017,382.67 bytes/sec [must-gather-8jssz] OUT 2025-12-29T00:26:12.915688694Z total size is 187,947,016 speedup is 6.95 [must-gather ] OUT 2025-12-29T00:26:12.943715461Z namespace/openshift-must-gather-qczlj deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.8 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing