[must-gather ] OUT 2025-11-25T00:17:35.441345452Z Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/authentication is not available (WellKnownAvailable: The well-known endpoint is not yet available: failed to get API server IPs: unable to find kube api server endpointLister port: &v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kubernetes", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9374e9d0-f290-4faa-94c7-262a199a1d45", ResourceVersion:"42871", Generation:0, CreationTimestamp:time.Date(2024, time.June, 26, 12, 38, 3, 0, time.Local), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/skip-mirror":"true"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-apiserver", Operation:"Update", APIVersion:"v1", Time:time.Date(2025, time.August, 13, 20, 8, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0029973f8), Subresource:""}}}, Subsets:[]v1.EndpointSubset(nil)} (check kube-apiserver that it deploys correctly)) because All is well clusteroperator/kube-apiserver is progressing: NodeInstallerProgressing: 1 node is at revision 12; 0 nodes have achieved new revision 13 clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing [must-gather ] OUT 2025-11-25T00:17:35.498729999Z namespace/openshift-must-gather-9xkx6 created [must-gather ] OUT 2025-11-25T00:17:35.501834286Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-9wrmd created [must-gather ] OUT 2025-11-25T00:17:35.68759036Z pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created [must-gather-c47g5] OUT 2025-11-25T00:19:09.815587779Z waiting for gather to complete [must-gather-c47g5] OUT 2025-11-25T00:19:09.817975637Z downloading gather output [must-gather-c47g5] OUT 2025-11-25T00:19:10.072819476Z receiving incremental file list [must-gather-c47g5] OUT 2025-11-25T00:19:10.079424349Z ./ [must-gather-c47g5] OUT 2025-11-25T00:19:10.079524741Z must-gather.tar.xz [must-gather-c47g5] OUT 2025-11-25T00:19:10.154098455Z packagemanifests [must-gather-c47g5] OUT 2025-11-25T00:19:10.929924609Z run.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.930034872Z version [must-gather-c47g5] OUT 2025-11-25T00:19:10.930073833Z crd/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930083183Z csv/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930091563Z ctlplane/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930153025Z ctlplane/endpoints [must-gather-c47g5] OUT 2025-11-25T00:19:10.930196256Z ctlplane/services [must-gather-c47g5] OUT 2025-11-25T00:19:10.930208686Z ctlplane/ovn/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930221346Z ctlplane/rabbitmq/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930229297Z dbs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930237247Z namespaces/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930245257Z namespaces/openshift-machine-api/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93037723Z namespaces/openshift-machine-api/all_resources.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.930399451Z namespaces/openshift-machine-api/events.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.930634636Z namespaces/openshift-machine-api/nad.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.930730519Z namespaces/openshift-machine-api/pvc.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.930765899Z namespaces/openshift-machine-api/buildconfig/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93077457Z namespaces/openshift-machine-api/cert/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93078333Z namespaces/openshift-machine-api/certificaterequests/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.9307989Z namespaces/openshift-machine-api/configmaps/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.9308074Z namespaces/openshift-machine-api/cronjobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930851022Z namespaces/openshift-machine-api/daemonset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930861582Z namespaces/openshift-machine-api/deployments/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.930878492Z namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.930945994Z namespaces/openshift-machine-api/deployments/machine-api-operator.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.931018686Z namespaces/openshift-machine-api/imagestream/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931030136Z namespaces/openshift-machine-api/installplans/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931050306Z namespaces/openshift-machine-api/issuer/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931061097Z namespaces/openshift-machine-api/jobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931071507Z namespaces/openshift-machine-api/pods/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931081917Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931099697Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/control-plane-machine-set-operator-649bd778b4-tt5tw-describe [must-gather-c47g5] OUT 2025-11-25T00:19:10.931224841Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931247022Z namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-649bd778b4-tt5tw/logs/control-plane-machine-set-operator.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.931382935Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931413666Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/machine-api-operator-788b7c6b6c-ctdmb-describe [must-gather-c47g5] OUT 2025-11-25T00:19:10.931557219Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93158238Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/kube-rbac-proxy.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.931710343Z namespaces/openshift-machine-api/pods/machine-api-operator-788b7c6b6c-ctdmb/logs/machine-api-operator.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.931811015Z namespaces/openshift-machine-api/replicaset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.931835926Z namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-649bd778b4.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.93199594Z namespaces/openshift-machine-api/replicaset/machine-api-operator-788b7c6b6c.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932032621Z namespaces/openshift-machine-api/routes/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.932043691Z namespaces/openshift-machine-api/services/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.932061711Z namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932199525Z namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932259836Z namespaces/openshift-machine-api/services/machine-api-controllers.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932375909Z namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932562913Z namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932657387Z namespaces/openshift-machine-api/services/machine-api-operator.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.932683087Z namespaces/openshift-machine-api/statefulset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.932693917Z namespaces/openshift-machine-api/subscriptions/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.932704508Z namespaces/openstack-operators/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.932766609Z namespaces/openstack-operators/all_resources.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.932852091Z namespaces/openstack-operators/events.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.932944523Z namespaces/openstack-operators/nad.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933057266Z namespaces/openstack-operators/pvc.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933081617Z namespaces/openstack-operators/buildconfig/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933090587Z namespaces/openstack-operators/cert/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933099357Z namespaces/openstack-operators/certificaterequests/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933107577Z namespaces/openstack-operators/configmaps/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933120708Z namespaces/openstack-operators/cronjobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933131408Z namespaces/openstack-operators/daemonset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933142178Z namespaces/openstack-operators/deployments/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933152578Z namespaces/openstack-operators/imagestream/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933162959Z namespaces/openstack-operators/installplans/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933173029Z namespaces/openstack-operators/issuer/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933182899Z namespaces/openstack-operators/jobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933193009Z namespaces/openstack-operators/pods/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93320992Z namespaces/openstack-operators/pods/-describe [must-gather-c47g5] OUT 2025-11-25T00:19:10.93322061Z namespaces/openstack-operators/pods/logs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93323049Z namespaces/openstack-operators/replicaset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933240801Z namespaces/openstack-operators/routes/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933250901Z namespaces/openstack-operators/services/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933260921Z namespaces/openstack-operators/statefulset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933270791Z namespaces/openstack-operators/subscriptions/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933280982Z namespaces/openstack/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933297742Z namespaces/openstack/all_resources.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933357653Z namespaces/openstack/events.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933468126Z namespaces/openstack/nad.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933536548Z namespaces/openstack/pvc.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933595079Z namespaces/openstack/buildconfig/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93361494Z namespaces/openstack/cert/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93362543Z namespaces/openstack/certificaterequests/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.93363596Z namespaces/openstack/configmaps/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933646Z namespaces/openstack/cronjobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933656431Z namespaces/openstack/daemonset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933666441Z namespaces/openstack/deployments/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933677381Z namespaces/openstack/imagestream/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933687421Z namespaces/openstack/installplans/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933699392Z namespaces/openstack/issuer/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933709392Z namespaces/openstack/jobs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933719522Z namespaces/openstack/pods/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933752113Z namespaces/openstack/pods/-describe [must-gather-c47g5] OUT 2025-11-25T00:19:10.933763223Z namespaces/openstack/pods/logs/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933773463Z namespaces/openstack/replicaset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933783644Z namespaces/openstack/routes/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933793804Z namespaces/openstack/services/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933803804Z namespaces/openstack/statefulset/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933813854Z namespaces/openstack/subscriptions/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933823925Z network/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933834205Z network/l2advertisement.log [must-gather-c47g5] OUT 2025-11-25T00:19:10.933851045Z network/ipaddresspools/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933863215Z network/nnce/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933874386Z network/nncp/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933885786Z network/nns/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933896086Z nodes/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.933906767Z nodes/crc.describe [must-gather-c47g5] OUT 2025-11-25T00:19:10.934066941Z nodes/crc.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.934232515Z sos-reports/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.934255676Z sos-reports/sos-report- -> ../_all_nodes/sosreport- [must-gather-c47g5] OUT 2025-11-25T00:19:10.934266316Z sos-reports/_all_nodes/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.934276726Z webhooks/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.934287157Z webhooks/mutating/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.934297627Z webhooks/validating/ [must-gather-c47g5] OUT 2025-11-25T00:19:10.934315737Z webhooks/validating/controlplanemachineset.machine.openshift.io.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.934408899Z webhooks/validating/multus.openshift.io.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.934510022Z webhooks/validating/network-node-identity.openshift.io.yaml [must-gather-c47g5] OUT 2025-11-25T00:19:10.938452019Z [must-gather-c47g5] OUT 2025-11-25T00:19:10.93848658Z sent 1,129 bytes received 27,231,454 bytes 18,155,055.33 bytes/sec [must-gather-c47g5] OUT 2025-11-25T00:19:10.93849834Z total size is 184,057,730 speedup is 6.76 [must-gather ] OUT 2025-11-25T00:19:10.966692884Z namespace/openshift-must-gather-9xkx6 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a84dabf3-edcf-4828-b6a1-f9d3a6f02304 ClientVersion: 4.20.3 ClusterVersion: Stable at "4.16.0" ClusterOperators: clusteroperator/machine-config is not upgradeable because One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/insights is missing clusteroperator/monitoring is missing clusteroperator/storage is missing