[must-gather ] OUT 2026-01-23T00:19:16.316821409Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42975", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0019a07f8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2026-01-23T00:19:16.364397092Z namespace/openshift-must-gather-wbthw created [must-gather ] OUT 2026-01-23T00:19:16.371638987Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-hckdb created [must-gather ] OUT 2026-01-23T00:19:16.379466999Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-jzld2] OUT 2026-01-23T00:21:23.370762692Z waiting for gather to complete [must-gather-jzld2] OUT 2026-01-23T00:21:23.376407915Z downloading gather output [must-gather-jzld2] OUT 2026-01-23T00:21:23.691420196Z receiving incremental file list [must-gather-jzld2] OUT 2026-01-23T00:21:23.697523901Z ./ [must-gather-jzld2] OUT 2026-01-23T00:21:23.697641794Z must-gather.tar.xz [must-gather-jzld2] OUT 2026-01-23T00:21:23.771256917Z packagemanifests [must-gather-jzld2] OUT 2026-01-23T00:21:24.584400697Z run.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.584555651Z version [must-gather-jzld2] OUT 2026-01-23T00:21:24.584616972Z crd/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584625053Z csv/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584630083Z ctlplane/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584712775Z ctlplane/endpoints [must-gather-jzld2] OUT 2026-01-23T00:21:24.584846049Z ctlplane/services [must-gather-jzld2] OUT 2026-01-23T00:21:24.584972372Z ctlplane/ovn/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584987162Z ctlplane/rabbitmq/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584992252Z dbs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.584997673Z namespaces/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585003563Z namespaces/openshift-machine-api/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585012923Z namespaces/openshift-machine-api/all_resources.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.585153517Z namespaces/openshift-machine-api/events.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.585347282Z namespaces/openshift-machine-api/nad.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.585506656Z namespaces/openshift-machine-api/pvc.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.585527437Z namespaces/openshift-machine-api/buildconfig/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585537017Z namespaces/openshift-machine-api/cert/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585542447Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585547087Z namespaces/openshift-machine-api/configmaps/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585551797Z namespaces/openshift-machine-api/cronjobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585556437Z namespaces/openshift-machine-api/daemonset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585560968Z namespaces/openshift-machine-api/deployments/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585569588Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.585735302Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.585816804Z namespaces/openshift-machine-api/imagestream/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585825005Z namespaces/openshift-machine-api/installplans/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585829895Z namespaces/openshift-machine-api/issuer/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585834725Z namespaces/openshift-machine-api/jobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585839365Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585847955Z namespaces/openshift-machine-api/pods/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.585852775Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58601623Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.58602958Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.586114482Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.586273327Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.586338428Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.586463912Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.586529333Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.586696498Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.586823952Z namespaces/openshift-machine-api/replicaset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.586879664Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.587004387Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.587064159Z namespaces/openshift-machine-api/routes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.587071629Z namespaces/openshift-machine-api/services/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.587162331Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.587365957Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.587509691Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.58784758Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.588094886Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.588261021Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.588329433Z namespaces/openshift-machine-api/statefulset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588337383Z namespaces/openshift-machine-api/subscriptions/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588342273Z namespaces/openshift-operators/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588410405Z namespaces/openshift-operators/all_resources.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.588543538Z namespaces/openshift-operators/events.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.5886067Z namespaces/openshift-operators/nad.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.588767654Z namespaces/openshift-operators/pvc.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.588776544Z namespaces/openshift-operators/buildconfig/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588784565Z namespaces/openshift-operators/cert/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588789125Z namespaces/openshift-operators/certificaterequests/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588793435Z namespaces/openshift-operators/configmaps/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588797755Z namespaces/openshift-operators/cronjobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588804915Z namespaces/openshift-operators/daemonset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588809205Z namespaces/openshift-operators/deployments/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588813475Z namespaces/openshift-operators/imagestream/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588817756Z namespaces/openshift-operators/installplans/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588821866Z namespaces/openshift-operators/issuer/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588825936Z namespaces/openshift-operators/jobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588832546Z namespaces/openshift-operators/poddisruptionbudgets/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588836756Z namespaces/openshift-operators/pods/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.588844006Z namespaces/openshift-operators/pods/-describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.589056842Z namespaces/openshift-operators/pods/logs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589068822Z namespaces/openshift-operators/replicaset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589073492Z namespaces/openshift-operators/routes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589077963Z namespaces/openshift-operators/services/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589082293Z namespaces/openshift-operators/statefulset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589086903Z namespaces/openshift-operators/subscriptions/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589091253Z namespaces/openstack-operators/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589095623Z namespaces/openstack-operators/all_resources.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.589100033Z namespaces/openstack-operators/events.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.589245977Z namespaces/openstack-operators/nad.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.589305989Z namespaces/openstack-operators/pvc.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.589313839Z namespaces/openstack-operators/buildconfig/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589318549Z namespaces/openstack-operators/cert/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589327399Z namespaces/openstack-operators/certificaterequests/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589332469Z namespaces/openstack-operators/configmaps/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589337139Z namespaces/openstack-operators/cronjobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58934171Z namespaces/openstack-operators/daemonset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58934611Z namespaces/openstack-operators/deployments/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58935085Z namespaces/openstack-operators/imagestream/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58936236Z namespaces/openstack-operators/installplans/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58937071Z namespaces/openstack-operators/issuer/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.58937573Z namespaces/openstack-operators/jobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589380441Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589385301Z namespaces/openstack-operators/pods/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.589393291Z namespaces/openstack-operators/pods/-describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.590381718Z namespaces/openstack-operators/pods/logs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590406239Z namespaces/openstack-operators/replicaset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590413399Z namespaces/openstack-operators/routes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590419429Z namespaces/openstack-operators/services/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.59042519Z namespaces/openstack-operators/statefulset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.5904309Z namespaces/openstack-operators/subscriptions/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.59043689Z namespaces/openstack/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.59044292Z namespaces/openstack/all_resources.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.5904488Z namespaces/openstack/events.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.59045447Z namespaces/openstack/nad.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.590462981Z namespaces/openstack/pvc.log [must-gather-jzld2] OUT 2026-01-23T00:21:24.590469041Z namespaces/openstack/buildconfig/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590474781Z namespaces/openstack/cert/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590480761Z namespaces/openstack/certificaterequests/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590486561Z namespaces/openstack/configmaps/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590492181Z namespaces/openstack/cronjobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590497792Z namespaces/openstack/daemonset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590503232Z namespaces/openstack/deployments/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590508792Z namespaces/openstack/imagestream/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590514312Z namespaces/openstack/installplans/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590519862Z namespaces/openstack/issuer/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590525352Z namespaces/openstack/jobs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590531022Z namespaces/openstack/poddisruptionbudgets/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590536513Z namespaces/openstack/pods/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590542743Z namespaces/openstack/pods/-describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.590548323Z namespaces/openstack/pods/logs/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590553773Z namespaces/openstack/replicaset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590559233Z namespaces/openstack/routes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590564763Z namespaces/openstack/services/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590570183Z namespaces/openstack/statefulset/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590575644Z namespaces/openstack/subscriptions/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590581534Z network/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590587474Z network/l2advertisement.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.590593084Z network/ipaddresspools/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590598724Z network/nnce/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590604394Z network/nncp/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590610185Z network/nns/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590615695Z nodes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590621375Z nodes/crc.describe [must-gather-jzld2] OUT 2026-01-23T00:21:24.590626905Z nodes/crc.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.590632465Z sos-reports/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590647026Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-jzld2] OUT 2026-01-23T00:21:24.590653006Z sos-reports/_all_nodes/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590658746Z webhooks/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590664476Z webhooks/mutating/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590670126Z webhooks/validating/ [must-gather-jzld2] OUT 2026-01-23T00:21:24.590675636Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.590688277Z webhooks/validating/multus.openshift.io.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.590694247Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-jzld2] OUT 2026-01-23T00:21:24.59452431Z [must-gather-jzld2] OUT 2026-01-23T00:21:24.594580092Z sent 1,316 bytes received 31,088,812 bytes 20,726,752.00 bytes/sec [must-gather-jzld2] OUT 2026-01-23T00:21:24.594588122Z total size is 213,361,001 speedup is 6.86 [must-gather ] OUT 2026-01-23T00:21:24.625995163Z namespace/openshift-must-gather-wbthw deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing