[must-gather ] OUT 2025-11-26T05:50:27.174108482Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42407", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000b996c8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-26T05:50:27.279574978Z namespace/openshift-must-gather-79cf2 created [must-gather ] OUT 2025-11-26T05:50:27.288833972Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-9hp4x created [must-gather ] OUT 2025-11-26T05:50:27.445255036Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-frqgs] OUT 2025-11-26T05:51:39.030878674Z waiting for gather to complete [must-gather-frqgs] OUT 2025-11-26T05:51:39.229423953Z downloading gather output [must-gather-frqgs] OUT 2025-11-26T05:51:39.90734855Z receiving incremental file list [must-gather-frqgs] OUT 2025-11-26T05:51:39.921497632Z ./ [must-gather-frqgs] OUT 2025-11-26T05:51:39.921848142Z must-gather.tar.xz [must-gather-frqgs] OUT 2025-11-26T05:51:40.001090127Z packagemanifests [must-gather-frqgs] OUT 2025-11-26T05:51:40.652475929Z run.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.652829029Z version [must-gather-frqgs] OUT 2025-11-26T05:51:40.65286208Z crd/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.652870181Z csv/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.652874171Z ctlplane/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.652887951Z ctlplane/endpoints [must-gather-frqgs] OUT 2025-11-26T05:51:40.653375925Z ctlplane/services [must-gather-frqgs] OUT 2025-11-26T05:51:40.653709715Z ctlplane/ovn/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.653727005Z ctlplane/rabbitmq/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.653732185Z dbs/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.653736785Z namespaces/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.653741075Z namespaces/openshift-machine-api/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.653745106Z namespaces/openshift-machine-api/all_resources.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.654317872Z namespaces/openshift-machine-api/events.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.654717603Z namespaces/openshift-machine-api/nad.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.654988271Z namespaces/openshift-machine-api/pvc.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.655181236Z namespaces/openshift-machine-api/buildconfig/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655191247Z namespaces/openshift-machine-api/cert/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655195337Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655200077Z namespaces/openshift-machine-api/configmaps/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655204477Z namespaces/openshift-machine-api/cronjobs/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655210117Z namespaces/openshift-machine-api/daemonset/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655214497Z namespaces/openshift-machine-api/deployments/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.655830645Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.656129623Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.65637645Z namespaces/openshift-machine-api/imagestream/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.656534095Z namespaces/openshift-machine-api/installplans/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.656680409Z namespaces/openshift-machine-api/issuer/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.65669635Z namespaces/openshift-machine-api/jobs/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.656906616Z namespaces/openshift-machine-api/pods/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.65705116Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.657274606Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-frqgs] OUT 2025-11-26T05:51:40.657541224Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.657734889Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.657971506Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.65813215Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-frqgs] OUT 2025-11-26T05:51:40.658339936Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.658525792Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.65882685Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.659067017Z namespaces/openshift-machine-api/replicaset/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.659263863Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.659506879Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.659753627Z namespaces/openshift-machine-api/routes/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.659765917Z namespaces/openshift-machine-api/services/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.660055845Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.660315573Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.660537939Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.660859018Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.661249669Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.661532497Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.661773484Z namespaces/openshift-machine-api/statefulset/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.661787814Z namespaces/openshift-machine-api/subscriptions/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.661792324Z network/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.662136844Z network/l2advertisement.log [must-gather-frqgs] OUT 2025-11-26T05:51:40.662800873Z network/ipaddresspools/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.662840334Z network/nnce/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.662872735Z network/nncp/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.662896566Z network/nns/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.662917797Z nodes/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.66303126Z nodes/crc.describe [must-gather-frqgs] OUT 2025-11-26T05:51:40.663359769Z nodes/crc.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.663546304Z sos-reports/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.663589886Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-frqgs] OUT 2025-11-26T05:51:40.663634957Z sos-reports/_all_nodes/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.663656658Z webhooks/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.663672678Z webhooks/mutating/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.663689489Z webhooks/validating/ [must-gather-frqgs] OUT 2025-11-26T05:51:40.663824102Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.663980817Z webhooks/validating/multus.openshift.io.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.664140211Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-frqgs] OUT 2025-11-26T05:51:40.666654203Z [must-gather-frqgs] OUT 2025-11-26T05:51:40.666764486Z sent 787 bytes received 26,351,944 bytes 17,568,487.33 bytes/sec [must-gather-frqgs] OUT 2025-11-26T05:51:40.666776636Z total size is 183,642,584 speedup is 6.97 [must-gather ] OUT 2025-11-26T05:51:40.694353021Z namespace/openshift-must-gather-79cf2 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.4 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing