[must-gather ] OUT 2026-01-28T06:48:37.533186231Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42482", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000453f08), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2026-01-28T06:48:37.578381755Z namespace/openshift-must-gather-p5tgt created [must-gather ] OUT 2026-01-28T06:48:37.582677316Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-7l9b5 created [must-gather ] OUT 2026-01-28T06:48:37.782039116Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-82hwg] OUT 2026-01-28T06:50:09.774161006Z waiting for gather to complete [must-gather-82hwg] OUT 2026-01-28T06:50:09.776867083Z downloading gather output [must-gather-82hwg] OUT 2026-01-28T06:50:10.078222869Z receiving incremental file list [must-gather-82hwg] OUT 2026-01-28T06:50:10.088379127Z ./ [must-gather-82hwg] OUT 2026-01-28T06:50:10.08955155Z must-gather.tar.xz [must-gather-82hwg] OUT 2026-01-28T06:50:10.176978807Z packagemanifests [must-gather-82hwg] OUT 2026-01-28T06:50:11.077932471Z run.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.078448696Z version [must-gather-82hwg] OUT 2026-01-28T06:50:11.078705533Z crd/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.078833906Z csv/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.07895131Z ctlplane/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079175456Z ctlplane/endpoints [must-gather-82hwg] OUT 2026-01-28T06:50:11.079235758Z ctlplane/services [must-gather-82hwg] OUT 2026-01-28T06:50:11.079495865Z ctlplane/ovn/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079536726Z ctlplane/rabbitmq/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079541996Z dbs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079552647Z namespaces/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079557397Z namespaces/openshift-machine-api/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.079997329Z namespaces/openshift-machine-api/all_resources.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.080239046Z namespaces/openshift-machine-api/events.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.080600157Z namespaces/openshift-machine-api/nad.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.080842783Z namespaces/openshift-machine-api/pvc.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.081026279Z namespaces/openshift-machine-api/buildconfig/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081148542Z namespaces/openshift-machine-api/cert/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081286026Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081305567Z namespaces/openshift-machine-api/configmaps/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081311087Z namespaces/openshift-machine-api/cronjobs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081315937Z namespaces/openshift-machine-api/daemonset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.081320217Z namespaces/openshift-machine-api/deployments/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.08179762Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.082064888Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.082277144Z namespaces/openshift-machine-api/imagestream/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.082420408Z namespaces/openshift-machine-api/installplans/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.082566802Z namespaces/openshift-machine-api/issuer/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.082701546Z namespaces/openshift-machine-api/jobs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.082824159Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.082936793Z namespaces/openshift-machine-api/pods/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.083018105Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.083245521Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-82hwg] OUT 2026-01-28T06:50:11.083421896Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.083646163Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.083879899Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.084062815Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-82hwg] OUT 2026-01-28T06:50:11.084278681Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.084470756Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.084741374Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.084935599Z namespaces/openshift-machine-api/replicaset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.085122145Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.085338611Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.085551967Z namespaces/openshift-machine-api/routes/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.085567837Z namespaces/openshift-machine-api/services/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.085947228Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.086180905Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.086418911Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.086684679Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.086989398Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.087240585Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.08741909Z namespaces/openshift-machine-api/statefulset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.087447921Z namespaces/openshift-machine-api/subscriptions/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.087452491Z namespaces/openshift-operators/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.08776943Z namespaces/openshift-operators/all_resources.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.088003396Z namespaces/openshift-operators/events.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.088215272Z namespaces/openshift-operators/nad.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.088445239Z namespaces/openshift-operators/pvc.log [must-gather-82hwg] OUT 2026-01-28T06:50:11.088647754Z namespaces/openshift-operators/buildconfig/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.088787938Z namespaces/openshift-operators/cert/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.088913632Z namespaces/openshift-operators/certificaterequests/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089033725Z namespaces/openshift-operators/configmaps/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089151609Z namespaces/openshift-operators/cronjobs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089284663Z namespaces/openshift-operators/daemonset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089393356Z namespaces/openshift-operators/deployments/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089575281Z namespaces/openshift-operators/imagestream/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089597481Z namespaces/openshift-operators/installplans/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089602062Z namespaces/openshift-operators/issuer/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089608032Z namespaces/openshift-operators/jobs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089612292Z namespaces/openshift-operators/poddisruptionbudgets/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.089616562Z namespaces/openshift-operators/pods/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090124566Z namespaces/openshift-operators/pods/-describe [must-gather-82hwg] OUT 2026-01-28T06:50:11.090312052Z namespaces/openshift-operators/pods/logs/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090448615Z namespaces/openshift-operators/replicaset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090476916Z namespaces/openshift-operators/routes/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090487037Z namespaces/openshift-operators/services/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090491547Z namespaces/openshift-operators/statefulset/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090497167Z namespaces/openshift-operators/subscriptions/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090502377Z network/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.090996311Z network/l2advertisement.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.091167356Z network/ipaddresspools/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.09131213Z network/nnce/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.091428513Z network/nncp/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.091587298Z network/nns/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.091710321Z nodes/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.091886536Z nodes/crc.describe [must-gather-82hwg] OUT 2026-01-28T06:50:11.092202565Z nodes/crc.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.092450942Z sos-reports/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.092480273Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-82hwg] OUT 2026-01-28T06:50:11.092491853Z sos-reports/_all_nodes/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.092499564Z webhooks/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.092511434Z webhooks/mutating/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.092515864Z webhooks/validating/ [must-gather-82hwg] OUT 2026-01-28T06:50:11.093031199Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.093302446Z webhooks/validating/multus.openshift.io.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.093552493Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-82hwg] OUT 2026-01-28T06:50:11.09695089Z [must-gather-82hwg] OUT 2026-01-28T06:50:11.09697032Z sent 962 bytes received 30,466,189 bytes 12,186,860.40 bytes/sec [must-gather-82hwg] OUT 2026-01-28T06:50:11.096986381Z total size is 213,039,713 speedup is 6.99 [must-gather ] OUT 2026-01-28T06:50:11.12486317Z namespace/openshift-must-gather-p5tgt deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.10 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/openshift-controller-manager is progressing: Progressing: deployment/route-controller-manager: observed generation is 17, desired generation is 18. clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing