[must-gather ] OUT 2025-10-09T01:15:52.519934639Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"47043", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000581e90), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-09T01:15:52.64549579Z namespace/openshift-must-gather-58pnr created [must-gather ] OUT 2025-10-09T01:15:52.650358411Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-49lbb created [must-gather ] OUT 2025-10-09T01:15:52.806003279Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-nskkc] OUT 2025-10-09T01:17:26.473197961Z waiting for gather to complete [must-gather-nskkc] OUT 2025-10-09T01:17:26.477874889Z downloading gather output [must-gather-nskkc] OUT 2025-10-09T01:17:26.815862198Z receiving incremental file list [must-gather-nskkc] OUT 2025-10-09T01:17:26.824131223Z ./ [must-gather-nskkc] OUT 2025-10-09T01:17:26.825028907Z must-gather.tar.xz [must-gather-nskkc] OUT 2025-10-09T01:17:26.895516484Z packagemanifests [must-gather-nskkc] OUT 2025-10-09T01:17:27.618862531Z run.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.619274252Z version [must-gather-nskkc] OUT 2025-10-09T01:17:27.619416095Z crd/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619424146Z csv/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619428536Z ctlplane/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619525438Z ctlplane/endpoints [must-gather-nskkc] OUT 2025-10-09T01:17:27.619707913Z ctlplane/services [must-gather-nskkc] OUT 2025-10-09T01:17:27.619781565Z ctlplane/ovn/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619791435Z ctlplane/rabbitmq/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619806286Z dbs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619829196Z namespaces/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.619833757Z namespaces/openshift-machine-api/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.61996831Z namespaces/openshift-machine-api/all_resources.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.620260978Z namespaces/openshift-machine-api/events.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.62068693Z namespaces/openshift-machine-api/nad.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.620939077Z namespaces/openshift-machine-api/pvc.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.621018409Z namespaces/openshift-machine-api/buildconfig/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62102627Z namespaces/openshift-machine-api/cert/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62103073Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.6210348Z namespaces/openshift-machine-api/configmaps/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62104068Z namespaces/openshift-machine-api/cronjobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62104455Z namespaces/openshift-machine-api/daemonset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62104859Z namespaces/openshift-machine-api/deployments/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621192874Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.621458301Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.621653296Z namespaces/openshift-machine-api/imagestream/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621659347Z namespaces/openshift-machine-api/installplans/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621663687Z namespaces/openshift-machine-api/issuer/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621673567Z namespaces/openshift-machine-api/jobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621678607Z namespaces/openshift-machine-api/pods/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.621683417Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62177564Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-nskkc] OUT 2025-10-09T01:17:27.62215213Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.622249732Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.622465988Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.622694444Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-nskkc] OUT 2025-10-09T01:17:27.622841388Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.622966092Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.623234899Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.623452416Z namespaces/openshift-machine-api/replicaset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.623588529Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.623801045Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.62400484Z namespaces/openshift-machine-api/routes/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.624020171Z namespaces/openshift-machine-api/services/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.624188835Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.624397001Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.624596506Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.624842753Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.625912092Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.626154278Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.626300513Z namespaces/openshift-machine-api/statefulset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.626319074Z namespaces/openshift-machine-api/subscriptions/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.626323134Z namespaces/openstack-operators/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.626420756Z namespaces/openstack-operators/all_resources.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.626605891Z namespaces/openstack-operators/events.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.626789786Z namespaces/openstack-operators/nad.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.627062343Z namespaces/openstack-operators/pvc.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.627153756Z namespaces/openstack-operators/buildconfig/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627191287Z namespaces/openstack-operators/cert/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627204597Z namespaces/openstack-operators/certificaterequests/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627208897Z namespaces/openstack-operators/configmaps/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627212828Z namespaces/openstack-operators/cronjobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627216838Z namespaces/openstack-operators/daemonset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627220798Z namespaces/openstack-operators/deployments/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627224778Z namespaces/openstack-operators/imagestream/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627228638Z namespaces/openstack-operators/installplans/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627232848Z namespaces/openstack-operators/issuer/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627236618Z namespaces/openstack-operators/jobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627243388Z namespaces/openstack-operators/pods/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62730402Z namespaces/openstack-operators/pods/-describe [must-gather-nskkc] OUT 2025-10-09T01:17:27.627401933Z namespaces/openstack-operators/pods/logs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627453854Z namespaces/openstack-operators/replicaset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627459164Z namespaces/openstack-operators/routes/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627473695Z namespaces/openstack-operators/services/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627478305Z namespaces/openstack-operators/statefulset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627488115Z namespaces/openstack-operators/subscriptions/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627492005Z namespaces/openstack/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.627551337Z namespaces/openstack/all_resources.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.627767992Z namespaces/openstack/events.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.627995669Z namespaces/openstack/nad.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.628221435Z namespaces/openstack/pvc.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.628319097Z namespaces/openstack/buildconfig/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628324277Z namespaces/openstack/cert/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628328067Z namespaces/openstack/certificaterequests/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628331868Z namespaces/openstack/configmaps/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628335648Z namespaces/openstack/cronjobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62843501Z namespaces/openstack/daemonset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628477381Z namespaces/openstack/deployments/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628487642Z namespaces/openstack/imagestream/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628497152Z namespaces/openstack/installplans/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628524633Z namespaces/openstack/issuer/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628533053Z namespaces/openstack/jobs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628543893Z namespaces/openstack/pods/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628691887Z namespaces/openstack/pods/-describe [must-gather-nskkc] OUT 2025-10-09T01:17:27.628720168Z namespaces/openstack/pods/logs/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628728798Z namespaces/openstack/replicaset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628761979Z namespaces/openstack/routes/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628770759Z namespaces/openstack/services/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62877926Z namespaces/openstack/statefulset/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62878783Z namespaces/openstack/subscriptions/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.62879664Z network/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628856752Z network/l2advertisement.log [must-gather-nskkc] OUT 2025-10-09T01:17:27.628939484Z network/ipaddresspools/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628952534Z network/nnce/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628961184Z network/nncp/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628969895Z network/nns/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.628993465Z nodes/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629040577Z nodes/crc.describe [must-gather-nskkc] OUT 2025-10-09T01:17:27.629391517Z nodes/crc.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.629599223Z sos-reports/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629613303Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-nskkc] OUT 2025-10-09T01:17:27.629620953Z sos-reports/_all_nodes/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629626233Z webhooks/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629630683Z webhooks/mutating/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629635044Z webhooks/validating/ [must-gather-nskkc] OUT 2025-10-09T01:17:27.629765637Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.630070845Z webhooks/validating/multus.openshift.io.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.630331732Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-nskkc] OUT 2025-10-09T01:17:27.635352509Z [must-gather-nskkc] OUT 2025-10-09T01:17:27.635372739Z sent 1,129 bytes received 25,830,108 bytes 17,220,824.67 bytes/sec [must-gather-nskkc] OUT 2025-10-09T01:17:27.63537803Z total size is 175,998,131 speedup is 6.81 [must-gather ] OUT 2025-10-09T01:17:27.666586148Z namespace/openshift-must-gather-58pnr deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.14 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing