[must-gather ] OUT 2025-10-01T21:57:30.392085512Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42577", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc001018f30), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-10-01T21:57:30.476785129Z namespace/openshift-must-gather-q94lw created [must-gather ] OUT 2025-10-01T21:57:30.480070271Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-6ssrc created [must-gather ] OUT 2025-10-01T21:57:30.666995078Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-vnm8r] OUT 2025-10-01T21:59:02.253784468Z waiting for gather to complete [must-gather-vnm8r] OUT 2025-10-01T21:59:02.257410539Z downloading gather output [must-gather-vnm8r] OUT 2025-10-01T21:59:02.64863344Z receiving incremental file list [must-gather-vnm8r] OUT 2025-10-01T21:59:02.659513452Z ./ [must-gather-vnm8r] OUT 2025-10-01T21:59:02.660812541Z must-gather.tar.xz [must-gather-vnm8r] OUT 2025-10-01T21:59:02.740160425Z packagemanifests [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392023359Z run.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392312798Z version [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392453063Z crd/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392462583Z csv/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392467093Z ctlplane/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392620508Z ctlplane/endpoints [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392793113Z ctlplane/services [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392945188Z ctlplane/ovn/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.392974539Z ctlplane/rabbitmq/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393090322Z dbs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393165575Z namespaces/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393246367Z namespaces/openshift-machine-api/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393370991Z namespaces/openshift-machine-api/all_resources.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393573677Z namespaces/openshift-machine-api/events.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.393929458Z namespaces/openshift-machine-api/nad.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394102694Z namespaces/openshift-machine-api/pvc.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394192816Z namespaces/openshift-machine-api/buildconfig/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394264908Z namespaces/openshift-machine-api/cert/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39434547Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394412392Z namespaces/openshift-machine-api/configmaps/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394483085Z namespaces/openshift-machine-api/cronjobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394539826Z namespaces/openshift-machine-api/daemonset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394627819Z namespaces/openshift-machine-api/deployments/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.394768143Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39499189Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395160845Z namespaces/openshift-machine-api/imagestream/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395250598Z namespaces/openshift-machine-api/installplans/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39531327Z namespaces/openshift-machine-api/issuer/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395381432Z namespaces/openshift-machine-api/jobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395444204Z namespaces/openshift-machine-api/pods/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395507706Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395665611Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-vnm8r] OUT 2025-10-01T21:59:03.395894138Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.396012962Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.396199778Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.396311781Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-vnm8r] OUT 2025-10-01T21:59:03.396493767Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.396690093Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39694062Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397133315Z namespaces/openshift-machine-api/replicaset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397312431Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397527828Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397707583Z namespaces/openshift-machine-api/routes/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397723164Z namespaces/openshift-machine-api/services/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.397851148Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.398046084Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39823731Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.398426375Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.398705914Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.39891298Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399083376Z namespaces/openshift-machine-api/statefulset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399093626Z namespaces/openshift-machine-api/subscriptions/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399098646Z namespaces/openstack-operators/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399246381Z namespaces/openstack-operators/all_resources.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399433707Z namespaces/openstack-operators/events.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399610572Z namespaces/openstack-operators/nad.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399837898Z namespaces/openstack-operators/pvc.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.399971672Z namespaces/openstack-operators/buildconfig/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400079815Z namespaces/openstack-operators/cert/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400094486Z namespaces/openstack-operators/certificaterequests/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400099886Z namespaces/openstack-operators/configmaps/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400107856Z namespaces/openstack-operators/cronjobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400113286Z namespaces/openstack-operators/daemonset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400118337Z namespaces/openstack-operators/deployments/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400123327Z namespaces/openstack-operators/imagestream/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400128577Z namespaces/openstack-operators/installplans/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400134277Z namespaces/openstack-operators/issuer/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400139287Z namespaces/openstack-operators/jobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400144347Z namespaces/openstack-operators/pods/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400324143Z namespaces/openstack-operators/pods/-describe [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400450997Z namespaces/openstack-operators/pods/logs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400482958Z namespaces/openstack-operators/replicaset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400580051Z namespaces/openstack-operators/routes/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400610382Z namespaces/openstack-operators/services/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400729386Z namespaces/openstack-operators/statefulset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.400826829Z namespaces/openstack-operators/subscriptions/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40086489Z namespaces/openstack/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401036755Z namespaces/openstack/all_resources.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40120176Z namespaces/openstack/events.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401373585Z namespaces/openstack/nad.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401538791Z namespaces/openstack/pvc.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401703046Z namespaces/openstack/buildconfig/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401716606Z namespaces/openstack/cert/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401722076Z namespaces/openstack/certificaterequests/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401726976Z namespaces/openstack/configmaps/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401731796Z namespaces/openstack/cronjobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401737377Z namespaces/openstack/daemonset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401749877Z namespaces/openstack/deployments/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401757777Z namespaces/openstack/imagestream/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401762817Z namespaces/openstack/installplans/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401783888Z namespaces/openstack/issuer/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401789328Z namespaces/openstack/jobs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.401794218Z namespaces/openstack/pods/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402018965Z namespaces/openstack/pods/-describe [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402151449Z namespaces/openstack/pods/logs/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40215978Z namespaces/openstack/replicaset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.4021661Z namespaces/openstack/routes/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40217082Z namespaces/openstack/services/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40217911Z namespaces/openstack/statefulset/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40218387Z namespaces/openstack/subscriptions/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402189231Z network/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402385827Z network/l2advertisement.log [must-gather-vnm8r] OUT 2025-10-01T21:59:03.40250299Z network/ipaddresspools/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402510561Z network/nnce/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402515071Z network/nncp/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402531231Z network/nns/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402535831Z nodes/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402630133Z nodes/crc.describe [must-gather-vnm8r] OUT 2025-10-01T21:59:03.402923162Z nodes/crc.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403080697Z sos-reports/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403093008Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403101178Z sos-reports/_all_nodes/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403106028Z webhooks/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403110858Z webhooks/mutating/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403115568Z webhooks/validating/ [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403207831Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403443718Z webhooks/validating/multus.openshift.io.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.403630944Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-vnm8r] OUT 2025-10-01T21:59:03.407351478Z [must-gather-vnm8r] OUT 2025-10-01T21:59:03.407365839Z sent 1,125 bytes received 25,414,839 bytes 16,943,976.00 bytes/sec [must-gather-vnm8r] OUT 2025-10-01T21:59:03.407370889Z total size is 175,239,616 speedup is 6.89 [must-gather ] OUT 2025-10-01T21:59:03.436838549Z namespace/openshift-must-gather-q94lw deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.19.13 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is degraded because Failed to resync 4.16.0 because: error required MachineConfigPool master is paused and cannot sync until it is unpaused clusteroperator/openshift-samples is degraded because Samples installation in error at 4.16.0: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing