[must-gather ] OUT 2026-01-29T08:51:31.641979424Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.11 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42938", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc002e44090), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2026-01-29T08:51:31.692458088Z namespace/openshift-must-gather-92xg9 created [must-gather ] OUT 2026-01-29T08:51:31.69743748Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-695zl created [must-gather ] OUT 2026-01-29T08:51:31.820447478Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-zx22s] OUT 2026-01-29T08:53:41.584825625Z waiting for gather to complete [must-gather-zx22s] OUT 2026-01-29T08:53:41.58849651Z downloading gather output [must-gather-zx22s] OUT 2026-01-29T08:53:41.836732448Z receiving incremental file list [must-gather-zx22s] OUT 2026-01-29T08:53:41.843237334Z ./ [must-gather-zx22s] OUT 2026-01-29T08:53:41.84415921Z must-gather.tar.xz [must-gather-zx22s] OUT 2026-01-29T08:53:41.923105109Z packagemanifests [must-gather-zx22s] OUT 2026-01-29T08:53:42.786351318Z run.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.786668777Z version [must-gather-zx22s] OUT 2026-01-29T08:53:42.786807691Z crd/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.786920574Z csv/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787014007Z ctlplane/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787253413Z ctlplane/endpoints [must-gather-zx22s] OUT 2026-01-29T08:53:42.787529221Z ctlplane/services [must-gather-zx22s] OUT 2026-01-29T08:53:42.787685566Z ctlplane/ovn/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787696216Z ctlplane/rabbitmq/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787700136Z dbs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787703996Z namespaces/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.787707866Z namespaces/openshift-machine-api/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.788144809Z namespaces/openshift-machine-api/all_resources.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.788390886Z namespaces/openshift-machine-api/events.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.788694885Z namespaces/openshift-machine-api/nad.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.788905661Z namespaces/openshift-machine-api/pvc.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.789080266Z namespaces/openshift-machine-api/buildconfig/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789094396Z namespaces/openshift-machine-api/cert/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789260741Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789271381Z namespaces/openshift-machine-api/configmaps/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789275571Z namespaces/openshift-machine-api/cronjobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789279231Z namespaces/openshift-machine-api/daemonset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789286622Z namespaces/openshift-machine-api/deployments/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.789633202Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.789902829Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.790126356Z namespaces/openshift-machine-api/imagestream/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790140586Z namespaces/openshift-machine-api/installplans/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790144826Z namespaces/openshift-machine-api/issuer/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790148736Z namespaces/openshift-machine-api/jobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790152426Z namespaces/openshift-machine-api/poddisruptionbudgets/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790155816Z namespaces/openshift-machine-api/pods/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790159657Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.790601159Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.790828576Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.79098428Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.791243918Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.791417243Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.791817514Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.791828554Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.791832454Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.792114412Z namespaces/openshift-machine-api/replicaset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.792313618Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.792577616Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.792764611Z namespaces/openshift-machine-api/routes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.792776191Z namespaces/openshift-machine-api/services/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.792996838Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.793264905Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.793494542Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.793706968Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.794010567Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.794238103Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.794453529Z namespaces/openshift-machine-api/statefulset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.794493451Z namespaces/openshift-machine-api/subscriptions/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.794522721Z namespaces/openshift-operators/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.794690526Z namespaces/openshift-operators/all_resources.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.794945893Z namespaces/openshift-operators/events.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.795109348Z namespaces/openshift-operators/nad.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.795293413Z namespaces/openshift-operators/pvc.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.795386966Z namespaces/openshift-operators/buildconfig/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795446728Z namespaces/openshift-operators/cert/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795471899Z namespaces/openshift-operators/certificaterequests/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795552841Z namespaces/openshift-operators/configmaps/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795623553Z namespaces/openshift-operators/cronjobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795661334Z namespaces/openshift-operators/daemonset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795719376Z namespaces/openshift-operators/deployments/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795787758Z namespaces/openshift-operators/imagestream/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795834869Z namespaces/openshift-operators/installplans/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.79587274Z namespaces/openshift-operators/issuer/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795924401Z namespaces/openshift-operators/jobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.795975533Z namespaces/openshift-operators/poddisruptionbudgets/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796005904Z namespaces/openshift-operators/pods/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796127007Z namespaces/openshift-operators/pods/-describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.79622784Z namespaces/openshift-operators/pods/logs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796286122Z namespaces/openshift-operators/replicaset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796369664Z namespaces/openshift-operators/routes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796411425Z namespaces/openshift-operators/services/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796454737Z namespaces/openshift-operators/statefulset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796502288Z namespaces/openshift-operators/subscriptions/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796527519Z namespaces/openstack-operators/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.796686173Z namespaces/openstack-operators/all_resources.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.796805487Z namespaces/openstack-operators/events.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.797023303Z namespaces/openstack-operators/nad.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.797237329Z namespaces/openstack-operators/pvc.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.797337932Z namespaces/openstack-operators/buildconfig/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797367103Z namespaces/openstack-operators/cert/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797408244Z namespaces/openstack-operators/certificaterequests/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797432795Z namespaces/openstack-operators/configmaps/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797469126Z namespaces/openstack-operators/cronjobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797509307Z namespaces/openstack-operators/daemonset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797538148Z namespaces/openstack-operators/deployments/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797569999Z namespaces/openstack-operators/imagestream/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797599499Z namespaces/openstack-operators/installplans/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.7976312Z namespaces/openstack-operators/issuer/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797670681Z namespaces/openstack-operators/jobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797695452Z namespaces/openstack-operators/poddisruptionbudgets/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797730133Z namespaces/openstack-operators/pods/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.797836356Z namespaces/openstack-operators/pods/-describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.797914758Z namespaces/openstack-operators/pods/logs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.79796674Z namespaces/openstack-operators/replicaset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798005581Z namespaces/openstack-operators/routes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798056172Z namespaces/openstack-operators/services/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798106884Z namespaces/openstack-operators/statefulset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798132995Z namespaces/openstack-operators/subscriptions/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798171436Z namespaces/openstack/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798255898Z namespaces/openstack/all_resources.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.798378282Z namespaces/openstack/events.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.798554777Z namespaces/openstack/nad.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.798734332Z namespaces/openstack/pvc.log [must-gather-zx22s] OUT 2026-01-29T08:53:42.798821034Z namespaces/openstack/buildconfig/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798868346Z namespaces/openstack/cert/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798892796Z namespaces/openstack/certificaterequests/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.798946918Z namespaces/openstack/configmaps/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.79900318Z namespaces/openstack/cronjobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799041661Z namespaces/openstack/daemonset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799080872Z namespaces/openstack/deployments/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799112743Z namespaces/openstack/imagestream/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799144604Z namespaces/openstack/installplans/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799186385Z namespaces/openstack/issuer/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799211086Z namespaces/openstack/jobs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799248687Z namespaces/openstack/poddisruptionbudgets/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799272197Z namespaces/openstack/pods/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799425152Z namespaces/openstack/pods/-describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.799504994Z namespaces/openstack/pods/logs/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799549085Z namespaces/openstack/replicaset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799573836Z namespaces/openstack/routes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799600907Z namespaces/openstack/services/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799633558Z namespaces/openstack/statefulset/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799668909Z namespaces/openstack/subscriptions/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799692769Z network/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799784222Z network/l2advertisement.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.799868254Z network/ipaddresspools/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799901225Z network/nnce/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799933006Z network/nncp/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.799971507Z network/nns/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800014708Z nodes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800124782Z nodes/crc.describe [must-gather-zx22s] OUT 2026-01-29T08:53:42.800381069Z nodes/crc.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.800577644Z sos-reports/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800624716Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-zx22s] OUT 2026-01-29T08:53:42.800654477Z sos-reports/_all_nodes/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800689258Z webhooks/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800724939Z webhooks/mutating/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800748949Z webhooks/validating/ [must-gather-zx22s] OUT 2026-01-29T08:53:42.800863813Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.801029257Z webhooks/validating/multus.openshift.io.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.801293835Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-zx22s] OUT 2026-01-29T08:53:42.80496592Z [must-gather-zx22s] OUT 2026-01-29T08:53:42.805037802Z sent 1,316 bytes received 30,792,385 bytes 20,529,134.00 bytes/sec [must-gather-zx22s] OUT 2026-01-29T08:53:42.805136565Z total size is 213,012,431 speedup is 6.92 [must-gather ] OUT 2026-01-29T08:53:42.834191386Z namespace/openshift-must-gather-92xg9 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.11 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing