[must-gather ] OUT 2025-11-27T00:18:42.522721248Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42905", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0023b0438), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-27T00:18:42.56877163Z namespace/openshift-must-gather-gzj56 created [must-gather ] OUT 2025-11-27T00:18:42.576677436Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-hgqhl created [must-gather ] OUT 2025-11-27T00:18:42.589762383Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-qk8tw] OUT 2025-11-27T00:20:26.195596457Z waiting for gather to complete [must-gather-qk8tw] OUT 2025-11-27T00:20:26.199477997Z downloading gather output [must-gather-qk8tw] OUT 2025-11-27T00:20:26.490750516Z receiving incremental file list [must-gather-qk8tw] OUT 2025-11-27T00:20:26.503361369Z ./ [must-gather-qk8tw] OUT 2025-11-27T00:20:26.503559135Z must-gather.tar.xz [must-gather-qk8tw] OUT 2025-11-27T00:20:26.557374869Z packagemanifests [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342602158Z run.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342674781Z version [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342689841Z crd/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342709682Z csv/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342717972Z ctlplane/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342732442Z ctlplane/endpoints [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342841976Z ctlplane/services [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342859236Z ctlplane/ovn/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342867767Z ctlplane/rabbitmq/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342876087Z dbs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342884477Z namespaces/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342901098Z namespaces/openshift-machine-api/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.342950429Z namespaces/openshift-machine-api/all_resources.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343245219Z namespaces/openshift-machine-api/events.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343538978Z namespaces/openshift-machine-api/nad.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34359845Z namespaces/openshift-machine-api/pvc.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343638271Z namespaces/openshift-machine-api/buildconfig/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343647251Z namespaces/openshift-machine-api/cert/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343660672Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343669892Z namespaces/openshift-machine-api/configmaps/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343678192Z namespaces/openshift-machine-api/cronjobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343686382Z namespaces/openshift-machine-api/daemonset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343694943Z namespaces/openshift-machine-api/deployments/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.343767065Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344031473Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344183248Z namespaces/openshift-machine-api/imagestream/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344193798Z namespaces/openshift-machine-api/installplans/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344202259Z namespaces/openshift-machine-api/issuer/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344210829Z namespaces/openshift-machine-api/jobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344218989Z namespaces/openshift-machine-api/pods/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344227179Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34424268Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344474017Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.344553819Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345327883Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345370474Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345568971Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345593321Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345828409Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345908991Z namespaces/openshift-machine-api/replicaset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.345993524Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34618229Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346249702Z namespaces/openshift-machine-api/routes/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346259392Z namespaces/openshift-machine-api/services/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346323234Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346522971Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346640764Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.346790249Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347054007Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34713098Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347229383Z namespaces/openshift-machine-api/statefulset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347239553Z namespaces/openshift-machine-api/subscriptions/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347248304Z namespaces/openstack-operators/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347315635Z namespaces/openstack-operators/all_resources.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347444529Z namespaces/openstack-operators/events.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347573703Z namespaces/openstack-operators/nad.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347713597Z namespaces/openstack-operators/pvc.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347734728Z namespaces/openstack-operators/buildconfig/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347746658Z namespaces/openstack-operators/cert/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347757069Z namespaces/openstack-operators/certificaterequests/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347774529Z namespaces/openstack-operators/configmaps/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347785419Z namespaces/openstack-operators/cronjobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34779518Z namespaces/openstack-operators/daemonset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34780537Z namespaces/openstack-operators/deployments/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34781678Z namespaces/openstack-operators/imagestream/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347826261Z namespaces/openstack-operators/installplans/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347836321Z namespaces/openstack-operators/issuer/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347845361Z namespaces/openstack-operators/jobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347853492Z namespaces/openstack-operators/pods/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347867562Z namespaces/openstack-operators/pods/-describe [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347875902Z namespaces/openstack-operators/pods/logs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347884103Z namespaces/openstack-operators/replicaset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347892243Z namespaces/openstack-operators/routes/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347900503Z namespaces/openstack-operators/services/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347908563Z namespaces/openstack-operators/statefulset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347916674Z namespaces/openstack-operators/subscriptions/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347930254Z namespaces/openstack/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.347938594Z namespaces/openstack/all_resources.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348082969Z namespaces/openstack/events.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348203413Z namespaces/openstack/nad.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348342737Z namespaces/openstack/pvc.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348369128Z namespaces/openstack/buildconfig/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348377898Z namespaces/openstack/cert/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348386228Z namespaces/openstack/certificaterequests/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348394449Z namespaces/openstack/configmaps/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348402579Z namespaces/openstack/cronjobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348410869Z namespaces/openstack/daemonset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348419069Z namespaces/openstack/deployments/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34842715Z namespaces/openstack/imagestream/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34843537Z namespaces/openstack/installplans/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.3484441Z namespaces/openstack/issuer/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34845242Z namespaces/openstack/jobs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348471681Z namespaces/openstack/pods/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348486131Z namespaces/openstack/pods/-describe [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348494812Z namespaces/openstack/pods/logs/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348508142Z namespaces/openstack/replicaset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348516502Z namespaces/openstack/routes/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348524733Z namespaces/openstack/services/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348532913Z namespaces/openstack/statefulset/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348541103Z namespaces/openstack/subscriptions/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348549473Z network/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348563254Z network/l2advertisement.log [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348615876Z network/ipaddresspools/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348625476Z network/nnce/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348634016Z network/nncp/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348642366Z network/nns/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348650647Z nodes/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348664297Z nodes/crc.describe [must-gather-qk8tw] OUT 2025-11-27T00:20:27.348899054Z nodes/crc.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349020018Z sos-reports/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349029819Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349038249Z sos-reports/_all_nodes/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349051819Z webhooks/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34906248Z webhooks/mutating/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.34907122Z webhooks/validating/ [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349115121Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.349987439Z webhooks/validating/multus.openshift.io.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.350067611Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-qk8tw] OUT 2025-11-27T00:20:27.352349382Z [must-gather-qk8tw] OUT 2025-11-27T00:20:27.352385583Z sent 1,133 bytes received 26,769,986 bytes 17,847,412.67 bytes/sec [must-gather-qk8tw] OUT 2025-11-27T00:20:27.352395123Z total size is 183,978,244 speedup is 6.87 [must-gather ] OUT 2025-11-27T00:20:27.381930351Z namespace/openshift-must-gather-gzj56 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing