[must-gather ] OUT 2025-10-07T00:17:04.223552583Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42953", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002731638), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-07T00:17:04.284650041Z namespace/openshift-must-gather-knbqz created [must-gather ] OUT 2025-10-07T00:17:04.287986Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-qlhmf created [must-gather ] OUT 2025-10-07T00:17:04.438098461Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-l47gk] OUT 2025-10-07T00:18:36.117957137Z waiting for gather to complete [must-gather-l47gk] OUT 2025-10-07T00:18:36.120737831Z downloading gather output [must-gather-l47gk] OUT 2025-10-07T00:18:36.416871923Z receiving incremental file list [must-gather-l47gk] OUT 2025-10-07T00:18:36.424857546Z ./ [must-gather-l47gk] OUT 2025-10-07T00:18:36.425065492Z must-gather.tar.xz [must-gather-l47gk] OUT 2025-10-07T00:18:36.501378174Z packagemanifests [must-gather-l47gk] OUT 2025-10-07T00:18:37.250227213Z run.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.250281895Z version [must-gather-l47gk] OUT 2025-10-07T00:18:37.25049195Z crd/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250512751Z csv/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250516981Z ctlplane/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250590953Z ctlplane/endpoints [must-gather-l47gk] OUT 2025-10-07T00:18:37.250603473Z ctlplane/services [must-gather-l47gk] OUT 2025-10-07T00:18:37.250608243Z ctlplane/ovn/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250612253Z ctlplane/rabbitmq/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250677695Z dbs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250682365Z namespaces/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250686765Z namespaces/openshift-machine-api/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.250761917Z namespaces/openshift-machine-api/all_resources.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.250771598Z namespaces/openshift-machine-api/events.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.250984953Z namespaces/openshift-machine-api/nad.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.251068536Z namespaces/openshift-machine-api/pvc.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.251124137Z namespaces/openshift-machine-api/buildconfig/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251130517Z namespaces/openshift-machine-api/cert/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251135067Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251138937Z namespaces/openshift-machine-api/configmaps/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251142808Z namespaces/openshift-machine-api/cronjobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251146718Z namespaces/openshift-machine-api/daemonset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251150748Z namespaces/openshift-machine-api/deployments/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.25124613Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.251317352Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.251393524Z namespaces/openshift-machine-api/imagestream/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251400634Z namespaces/openshift-machine-api/installplans/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251404605Z namespaces/openshift-machine-api/issuer/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251408545Z namespaces/openshift-machine-api/jobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251420335Z namespaces/openshift-machine-api/pods/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251424465Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251547138Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-l47gk] OUT 2025-10-07T00:18:37.25161177Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251667051Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.251771714Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251822636Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-l47gk] OUT 2025-10-07T00:18:37.251916278Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.251965699Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.252125104Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.252207706Z namespaces/openshift-machine-api/replicaset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.252252457Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.25235821Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.252435432Z namespaces/openshift-machine-api/routes/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.252455562Z namespaces/openshift-machine-api/services/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.252513964Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.252620137Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.25272831Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.252819002Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.253002427Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.25311184Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.253172712Z namespaces/openshift-machine-api/statefulset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253178912Z namespaces/openshift-machine-api/subscriptions/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253185112Z namespaces/openstack-operators/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253242513Z namespaces/openstack-operators/all_resources.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253312595Z namespaces/openstack-operators/events.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253419278Z namespaces/openstack-operators/nad.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253561872Z namespaces/openstack-operators/pvc.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253614323Z namespaces/openstack-operators/buildconfig/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253619964Z namespaces/openstack-operators/cert/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253624064Z namespaces/openstack-operators/certificaterequests/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253628374Z namespaces/openstack-operators/configmaps/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253632394Z namespaces/openstack-operators/cronjobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253636454Z namespaces/openstack-operators/daemonset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253640524Z namespaces/openstack-operators/deployments/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253644394Z namespaces/openstack-operators/imagestream/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253648144Z namespaces/openstack-operators/installplans/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253651904Z namespaces/openstack-operators/issuer/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253655634Z namespaces/openstack-operators/jobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253659535Z namespaces/openstack-operators/pods/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253666115Z namespaces/openstack-operators/pods/-describe [must-gather-l47gk] OUT 2025-10-07T00:18:37.253799858Z namespaces/openstack-operators/pods/logs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253805818Z namespaces/openstack-operators/replicaset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253809559Z namespaces/openstack-operators/routes/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253813229Z namespaces/openstack-operators/services/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253816979Z namespaces/openstack-operators/statefulset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253821039Z namespaces/openstack-operators/subscriptions/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253826619Z namespaces/openstack/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.253831049Z namespaces/openstack/all_resources.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253951722Z namespaces/openstack/events.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.253966793Z namespaces/openstack/nad.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.254066125Z namespaces/openstack/pvc.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.254162308Z namespaces/openstack/buildconfig/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254174528Z namespaces/openstack/cert/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254178868Z namespaces/openstack/certificaterequests/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254183098Z namespaces/openstack/configmaps/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254187329Z namespaces/openstack/cronjobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254191929Z namespaces/openstack/daemonset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254196319Z namespaces/openstack/deployments/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254302392Z namespaces/openstack/imagestream/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254307282Z namespaces/openstack/installplans/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254315032Z namespaces/openstack/issuer/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254319242Z namespaces/openstack/jobs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254399894Z namespaces/openstack/pods/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254451726Z namespaces/openstack/pods/-describe [must-gather-l47gk] OUT 2025-10-07T00:18:37.254532408Z namespaces/openstack/pods/logs/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254548608Z namespaces/openstack/replicaset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254552958Z namespaces/openstack/routes/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254557618Z namespaces/openstack/services/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254561709Z namespaces/openstack/statefulset/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254637651Z namespaces/openstack/subscriptions/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254641891Z network/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254647261Z network/l2advertisement.log [must-gather-l47gk] OUT 2025-10-07T00:18:37.254651931Z network/ipaddresspools/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254737163Z network/nnce/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254742013Z network/nncp/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254746714Z network/nns/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254814805Z nodes/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254820395Z nodes/crc.describe [must-gather-l47gk] OUT 2025-10-07T00:18:37.254885367Z nodes/crc.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.254896638Z sos-reports/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254900728Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-l47gk] OUT 2025-10-07T00:18:37.254905108Z sos-reports/_all_nodes/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.254909348Z webhooks/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.2549856Z webhooks/mutating/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.25499054Z webhooks/validating/ [must-gather-l47gk] OUT 2025-10-07T00:18:37.2549949Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.255070502Z webhooks/validating/multus.openshift.io.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.255082832Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-l47gk] OUT 2025-10-07T00:18:37.258236507Z [must-gather-l47gk] OUT 2025-10-07T00:18:37.258256367Z sent 1,129 bytes received 25,125,479 bytes 16,751,072.00 bytes/sec [must-gather-l47gk] OUT 2025-10-07T00:18:37.258274848Z total size is 175,067,529 speedup is 6.97 [must-gather ] OUT 2025-10-07T00:18:37.291050621Z namespace/openshift-must-gather-knbqz deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing