[must-gather ] OUT 2025-10-08T00:17:49.392208035Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43025", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc001ef8750), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-08T00:17:49.453387838Z namespace/openshift-must-gather-r9nln created [must-gather ] OUT 2025-10-08T00:17:49.461374108Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-b5q7c created [must-gather ] OUT 2025-10-08T00:17:49.620393289Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-8mswm] OUT 2025-10-08T00:19:20.660280669Z waiting for gather to complete [must-gather-8mswm] OUT 2025-10-08T00:19:20.662919583Z downloading gather output [must-gather-8mswm] OUT 2025-10-08T00:19:20.927133198Z receiving incremental file list [must-gather-8mswm] OUT 2025-10-08T00:19:20.933592765Z ./ [must-gather-8mswm] OUT 2025-10-08T00:19:20.933756451Z must-gather.tar.xz [must-gather-8mswm] OUT 2025-10-08T00:19:20.98711297Z packagemanifests [must-gather-8mswm] OUT 2025-10-08T00:19:21.7371358Z run.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.737352187Z version [must-gather-8mswm] OUT 2025-10-08T00:19:21.737413239Z crd/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737423639Z csv/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737428139Z ctlplane/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737532142Z ctlplane/endpoints [must-gather-8mswm] OUT 2025-10-08T00:19:21.737713747Z ctlplane/services [must-gather-8mswm] OUT 2025-10-08T00:19:21.737747738Z ctlplane/ovn/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737758639Z ctlplane/rabbitmq/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737773559Z dbs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.73778411Z namespaces/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.73779413Z namespaces/openshift-machine-api/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.737874533Z namespaces/openshift-machine-api/all_resources.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.738118841Z namespaces/openshift-machine-api/events.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.73840944Z namespaces/openshift-machine-api/nad.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.738545664Z namespaces/openshift-machine-api/pvc.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.738599696Z namespaces/openshift-machine-api/buildconfig/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738612917Z namespaces/openshift-machine-api/cert/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738628347Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738637957Z namespaces/openshift-machine-api/configmaps/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738647798Z namespaces/openshift-machine-api/cronjobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738663858Z namespaces/openshift-machine-api/daemonset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738673008Z namespaces/openshift-machine-api/deployments/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.738693569Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.739004379Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.739078202Z namespaces/openshift-machine-api/imagestream/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739100902Z namespaces/openshift-machine-api/installplans/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739111113Z namespaces/openshift-machine-api/issuer/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739120833Z namespaces/openshift-machine-api/jobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739130203Z namespaces/openshift-machine-api/pods/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739145304Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739210416Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-8mswm] OUT 2025-10-08T00:19:21.739452164Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739590288Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.739758864Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.739862027Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-8mswm] OUT 2025-10-08T00:19:21.740043683Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.740122946Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.740356932Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.740549278Z namespaces/openshift-machine-api/replicaset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.74060165Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.740796636Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.74091901Z namespaces/openshift-machine-api/routes/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.740927721Z namespaces/openshift-machine-api/services/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.741037594Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741167468Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741281932Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741419897Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741655114Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741811999Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.741884562Z namespaces/openshift-machine-api/statefulset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.741891942Z namespaces/openshift-machine-api/subscriptions/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.741896082Z namespaces/openstack-operators/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.741981745Z namespaces/openstack-operators/all_resources.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742121119Z namespaces/openstack-operators/events.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742272874Z namespaces/openstack-operators/nad.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742422979Z namespaces/openstack-operators/pvc.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742481411Z namespaces/openstack-operators/buildconfig/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742488291Z namespaces/openstack-operators/cert/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742493341Z namespaces/openstack-operators/certificaterequests/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742497721Z namespaces/openstack-operators/configmaps/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742502252Z namespaces/openstack-operators/cronjobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742506272Z namespaces/openstack-operators/daemonset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742529992Z namespaces/openstack-operators/deployments/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742546263Z namespaces/openstack-operators/imagestream/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742552343Z namespaces/openstack-operators/installplans/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742557383Z namespaces/openstack-operators/issuer/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742562944Z namespaces/openstack-operators/jobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742568444Z namespaces/openstack-operators/pods/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742675717Z namespaces/openstack-operators/pods/-describe [must-gather-8mswm] OUT 2025-10-08T00:19:21.742690528Z namespaces/openstack-operators/pods/logs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742697018Z namespaces/openstack-operators/replicaset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742702448Z namespaces/openstack-operators/routes/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742706978Z namespaces/openstack-operators/services/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742720699Z namespaces/openstack-operators/statefulset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742725709Z namespaces/openstack-operators/subscriptions/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742730629Z namespaces/openstack/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.742746999Z namespaces/openstack/all_resources.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742869113Z namespaces/openstack/events.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.742989337Z namespaces/openstack/nad.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.743119401Z namespaces/openstack/pvc.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.743166332Z namespaces/openstack/buildconfig/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743173552Z namespaces/openstack/cert/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743178172Z namespaces/openstack/certificaterequests/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743182083Z namespaces/openstack/configmaps/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743186283Z namespaces/openstack/cronjobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743190733Z namespaces/openstack/daemonset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743194643Z namespaces/openstack/deployments/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743202033Z namespaces/openstack/imagestream/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743206093Z namespaces/openstack/installplans/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743210323Z namespaces/openstack/issuer/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743214424Z namespaces/openstack/jobs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743244635Z namespaces/openstack/pods/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743301506Z namespaces/openstack/pods/-describe [must-gather-8mswm] OUT 2025-10-08T00:19:21.743351698Z namespaces/openstack/pods/logs/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743362368Z namespaces/openstack/replicaset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743366699Z namespaces/openstack/routes/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743371429Z namespaces/openstack/services/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743376039Z namespaces/openstack/statefulset/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743380449Z namespaces/openstack/subscriptions/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743388319Z network/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743418Z network/l2advertisement.log [must-gather-8mswm] OUT 2025-10-08T00:19:21.743492743Z network/ipaddresspools/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743502383Z network/nnce/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743507283Z network/nncp/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743511253Z network/nns/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743515403Z nodes/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743611627Z nodes/crc.describe [must-gather-8mswm] OUT 2025-10-08T00:19:21.743799313Z nodes/crc.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.743971358Z sos-reports/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743978218Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-8mswm] OUT 2025-10-08T00:19:21.743982539Z sos-reports/_all_nodes/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743986349Z webhooks/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743993859Z webhooks/mutating/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.743997919Z webhooks/validating/ [must-gather-8mswm] OUT 2025-10-08T00:19:21.744099682Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.744276718Z webhooks/validating/multus.openshift.io.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.744431773Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-8mswm] OUT 2025-10-08T00:19:21.747787041Z [must-gather-8mswm] OUT 2025-10-08T00:19:21.747802131Z sent 1,129 bytes received 26,186,671 bytes 17,458,533.33 bytes/sec [must-gather-8mswm] OUT 2025-10-08T00:19:21.747806671Z total size is 176,745,290 speedup is 6.75 [must-gather ] OUT 2025-10-08T00:19:21.775587691Z namespace/openshift-must-gather-r9nln deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing