[must-gather ] OUT 2025-11-27T00:17:11.793823928Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"43055", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc004081020), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-27T00:17:11.858207825Z namespace/openshift-must-gather-jdfrz created [must-gather ] OUT 2025-11-27T00:17:11.863339855Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-cnkg4 created [must-gather ] OUT 2025-11-27T00:17:11.982070938Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-4pqn5] OUT 2025-11-27T00:18:48.643195484Z waiting for gather to complete [must-gather-4pqn5] OUT 2025-11-27T00:18:48.649024255Z downloading gather output [must-gather-4pqn5] OUT 2025-11-27T00:18:48.929417225Z receiving incremental file list [must-gather-4pqn5] OUT 2025-11-27T00:18:48.940044895Z ./ [must-gather-4pqn5] OUT 2025-11-27T00:18:48.940153958Z must-gather.tar.xz [must-gather-4pqn5] OUT 2025-11-27T00:18:49.003845164Z packagemanifests [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759235831Z run.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759304793Z version [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759312563Z crd/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759344234Z csv/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759362245Z ctlplane/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759373945Z ctlplane/endpoints [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759552191Z ctlplane/services [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759569771Z ctlplane/ovn/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759575372Z ctlplane/rabbitmq/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759580242Z dbs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759585262Z namespaces/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759590192Z namespaces/openshift-machine-api/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759595372Z namespaces/openshift-machine-api/all_resources.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759715206Z namespaces/openshift-machine-api/events.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759876421Z namespaces/openshift-machine-api/nad.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.759974634Z namespaces/openshift-machine-api/pvc.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760022966Z namespaces/openshift-machine-api/buildconfig/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760028876Z namespaces/openshift-machine-api/cert/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760033626Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760038496Z namespaces/openshift-machine-api/configmaps/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760043516Z namespaces/openshift-machine-api/cronjobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760139209Z namespaces/openshift-machine-api/daemonset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76015008Z namespaces/openshift-machine-api/deployments/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76016119Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760277784Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760299244Z namespaces/openshift-machine-api/imagestream/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760309245Z namespaces/openshift-machine-api/installplans/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760314405Z namespaces/openshift-machine-api/issuer/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760318955Z namespaces/openshift-machine-api/jobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760323765Z namespaces/openshift-machine-api/pods/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760331975Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760342226Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76046981Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76048538Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760627055Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760666166Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760781819Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760822011Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.760916114Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761059608Z namespaces/openshift-machine-api/replicaset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761083069Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761192611Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761277474Z namespaces/openshift-machine-api/routes/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761285314Z namespaces/openshift-machine-api/services/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761289904Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761391448Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76145954Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761542112Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761737588Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761818331Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761874273Z namespaces/openshift-machine-api/statefulset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761881523Z namespaces/openshift-machine-api/subscriptions/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761885363Z namespaces/openstack-operators/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.761935045Z namespaces/openstack-operators/all_resources.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762012757Z namespaces/openstack-operators/events.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762082499Z namespaces/openstack-operators/nad.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762173302Z namespaces/openstack-operators/pvc.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762194643Z namespaces/openstack-operators/buildconfig/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762198993Z namespaces/openstack-operators/cert/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762202673Z namespaces/openstack-operators/certificaterequests/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762209163Z namespaces/openstack-operators/configmaps/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762213113Z namespaces/openstack-operators/cronjobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762217123Z namespaces/openstack-operators/daemonset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762222744Z namespaces/openstack-operators/deployments/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762227264Z namespaces/openstack-operators/imagestream/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762230994Z namespaces/openstack-operators/installplans/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762234874Z namespaces/openstack-operators/issuer/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762238724Z namespaces/openstack-operators/jobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762242734Z namespaces/openstack-operators/pods/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762249645Z namespaces/openstack-operators/pods/-describe [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762303366Z namespaces/openstack-operators/pods/logs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762308096Z namespaces/openstack-operators/replicaset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762312206Z namespaces/openstack-operators/routes/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762315867Z namespaces/openstack-operators/services/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762319507Z namespaces/openstack-operators/statefulset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762323127Z namespaces/openstack-operators/subscriptions/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762326757Z namespaces/openstack/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762353938Z namespaces/openstack/all_resources.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.7624297Z namespaces/openstack/events.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762496852Z namespaces/openstack/nad.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762584705Z namespaces/openstack/pvc.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762604726Z namespaces/openstack/buildconfig/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762608916Z namespaces/openstack/cert/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762612856Z namespaces/openstack/certificaterequests/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762619316Z namespaces/openstack/configmaps/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762623256Z namespaces/openstack/cronjobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762627096Z namespaces/openstack/daemonset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762630906Z namespaces/openstack/deployments/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762634537Z namespaces/openstack/imagestream/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762638517Z namespaces/openstack/installplans/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762642157Z namespaces/openstack/issuer/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762646387Z namespaces/openstack/jobs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762656517Z namespaces/openstack/pods/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762678448Z namespaces/openstack/pods/-describe [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762724729Z namespaces/openstack/pods/logs/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76272914Z namespaces/openstack/replicaset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.7627328Z namespaces/openstack/routes/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76273704Z namespaces/openstack/services/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76274089Z namespaces/openstack/statefulset/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76274453Z namespaces/openstack/subscriptions/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76274849Z network/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762778411Z network/l2advertisement.log [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762817122Z network/ipaddresspools/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762821432Z network/nnce/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762825413Z network/nncp/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762835523Z network/nns/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762839243Z nodes/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.762859724Z nodes/crc.describe [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763103821Z nodes/crc.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763205555Z sos-reports/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763213265Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763217225Z sos-reports/_all_nodes/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763221615Z webhooks/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763225475Z webhooks/mutating/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763229615Z webhooks/validating/ [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763248186Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.76336508Z webhooks/validating/multus.openshift.io.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.763457882Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-4pqn5] OUT 2025-11-27T00:18:49.766649702Z [must-gather-4pqn5] OUT 2025-11-27T00:18:49.766669862Z sent 1,129 bytes received 26,944,582 bytes 17,963,807.33 bytes/sec [must-gather-4pqn5] OUT 2025-11-27T00:18:49.766677002Z total size is 184,000,324 speedup is 6.83 [must-gather ] OUT 2025-11-27T00:18:49.794479725Z namespace/openshift-must-gather-jdfrz deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing