--- I0319 12:15:50.158985 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0319 12:15:50.162073 11 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0319 12:15:50.162147 11 controller.go:128] Shutting down kubernetes service endpoint reconciler I0319 12:15:50.162285 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0319 12:15:50.692282 11 trace.go:236] Trace[1954376464]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:3c9f0a08-5937-4921-bc9c-09be46486f14,client:192.168.32.10,api-group:rbac.authorization.k8s.io,api-version:v1,name:openshift-marketplace-metrics,subresource:,namespace:openshift-marketplace,protocol:HTTP/2.0,resource:roles,scope:resource,url:/apis/rbac.authorization.k8s.io/v1/namespaces/openshift-marketplace/roles/openshift-marketplace-metrics,user-agent:cluster-version-operator/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (19-Mar-2026 12:15:50.156) (total time: 535ms): Trace[1954376464]: ---"About to write a response" 535ms (12:15:50.692) Trace[1954376464]: [535.477385ms] [535.477385ms] END I0319 12:15:50.693606 11 trace.go:236] Trace[1547446184]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:26cd1b27-9367-41cf-9424-7945f75be184,client:192.168.32.10,api-group:,api-version:v1,name:,subresource:,namespace:openshift-kube-apiserver,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/openshift-kube-apiserver/events,user-agent:kubelet/v1.31.14 (linux/amd64) kubernetes/8311c4d,verb:POST (19-Mar-2026 12:15:50.146) (total time: 546ms): Trace[1547446184]: ["Create etcd3" audit-id:26cd1b27-9367-41cf-9424-7945f75be184,key:/events/openshift-kube-apiserver/kube-apiserver-master-0.189e3d29247860d4,type:*core.Event,resource:events 545ms (12:15:50.147) Trace[1547446184]: ---"Txn call succeeded" 545ms (12:15:50.693)] Trace[1547446184]: [546.845861ms] [546.845861ms] END I0319 12:15:50.694687 11 trace.go:236] Trace[1537449944]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:36dfe376-2a8d-43bd-8e96-918dcc5dbfd6,client:10.128.0.18,api-group:,api-version:v1,name:etcd-serving-master-0,subresource:,namespace:openshift-etcd,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/openshift-etcd/secrets/etcd-serving-master-0,user-agent:cluster-etcd-operator/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (19-Mar-2026 12:15:50.162) (total time: 532ms): Trace[1537449944]: ---"About to write a response" 532ms (12:15:50.694) Trace[1537449944]: [532.500622ms] [532.500622ms] END I0319 12:15:50.695915 11 trace.go:236] Trace[297236611]: "List" accept:application/vnd.kubernetes.protobuf, */*,audit-id:b7b08eac-b3f0-435c-ad0c-73d44f97acf7,client:::1,api-group:,api-version:v1,name:,subresource:,namespace:openshift-kube-apiserver,protocol:HTTP/2.0,resource:limitranges,scope:namespace,url:/api/v1/namespaces/openshift-kube-apiserver/limitranges,user-agent:kube-apiserver/v1.31.14 (linux/amd64) kubernetes/8311c4d,verb:LIST (19-Mar-2026 12:15:50.180) (total time: 515ms): Trace[297236611]: ["List(recursive=false) etcd3" audit-id:b7b08eac-b3f0-435c-ad0c-73d44f97acf7,key:/limitranges,resourceVersion:,resourceVersionMatch:,limit:1,continue: 515ms (12:15:50.180)] Trace[297236611]: [515.820781ms] [515.820781ms] END I0319 12:15:50.699022 11 trace.go:236] Trace[1919400816]: "Update" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:fe4c7c67-7a92-4192-aaed-7693033673ba,client:::1,api-group:coordination.k8s.io,api-version:v1,name:cert-regeneration-controller-lock,subresource:,namespace:openshift-kube-apiserver,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/openshift-kube-apiserver/leases/cert-regeneration-controller-lock,user-agent:cluster-kube-apiserver-operator/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (19-Mar-2026 12:15:50.163) (total time: 535ms): Trace[1919400816]: ["GuaranteedUpdate etcd3" audit-id:fe4c7c67-7a92-4192-aaed-7693033673ba,key:/leases/openshift-kube-apiserver/cert-regeneration-controller-lock,type:*coordination.Lease,resource:leases.coordination.k8s.io 535ms (12:15:50.163) Trace[1919400816]: ---"Txn call completed" 534ms (12:15:50.698)] Trace[1919400816]: [535.500895ms] [535.500895ms] END I0319 12:15:50.699697 11 trace.go:236] Trace[424352117]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:3aba4808-0168-451f-a08b-009275d9e1fd,client:192.168.32.10,api-group:apiextensions.k8s.io,api-version:v1,name:machinehealthchecks.machine.openshift.io,subresource:,namespace:,protocol:HTTP/2.0,resource:customresourcedefinitions,scope:resource,url:/apis/apiextensions.k8s.io/v1/customresourcedefinitions/machinehealthchecks.machine.openshift.io,user-agent:cluster-version-operator/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (19-Mar-2026 12:15:50.182) (total time: 517ms): Trace[424352117]: ---"About to write a response" 517ms (12:15:50.699) Trace[424352117]: [517.39753ms] [517.39753ms] END I0319 12:15:51.112919 11 trace.go:236] Trace[2029361996]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:a0ee58da-14c0-4fdc-85af-ba039cb68839,client:192.168.32.10,api-group:,api-version:v1,name:,subresource:,namespace:openshift-kube-apiserver,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/openshift-kube-apiserver/pods,user-agent:kubelet/v1.31.14 (linux/amd64) kubernetes/8311c4d,verb:POST (19-Mar-2026 12:15:50.177) (total time: 934ms): Trace[2029361996]: [934.860939ms] [934.860939ms] END I0319 12:15:51.113249 11 trace.go:236] Trace[1530092368]: "Create etcd3" audit-id:,key:/events/openshift-kube-apiserver/kube-apiserver-master-0.189e3d292577a385,type:*core.Event,resource:events (19-Mar-2026 12:15:50.162) (total time: 950ms): Trace[1530092368]: ---"TransformToStorage succeeded" 535ms (12:15:50.698) Trace[1530092368]: ---"Txn call succeeded" 414ms (12:15:51.113) Trace[1530092368]: [950.832668ms] [950.832668ms] END I0319 12:15:51.113321 11 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0319 12:15:51.113347 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0319 12:15:51.129052 11 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0319 12:15:51.145676 11 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0319 12:15:51.145712 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0319 12:15:51.145958 11 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0319 12:15:51.145975 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0319 12:15:51.146088 11 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0319 12:15:51.150940 11 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0319 12:15:51.151067 11 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0319 12:15:51.151180 11 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 12:15:51.151221 11 controller.go:157] Shutting down quota evaluator I0319 12:15:51.151258 11 controller.go:176] quota evaluator worker shutdown I0319 12:15:51.151556 11 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0319 12:15:51.152035 11 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0319 12:15:51.152057 11 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0319 12:15:51.152098 11 controller.go:120] Shutting down OpenAPI V3 controller I0319 12:15:51.152094 11 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 12:15:51.152109 11 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0319 12:15:51.152116 11 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0319 12:15:51.152132 11 establishing_controller.go:92] Shutting down EstablishingController I0319 12:15:51.152158 11 crd_finalizer.go:281] Shutting down CRDFinalizer I0319 12:15:51.152176 11 naming_controller.go:305] Shutting down NamingConditionController I0319 12:15:51.152189 11 controller.go:170] Shutting down OpenAPI controller I0319 12:15:51.152203 11 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0319 12:15:51.152206 11 controller.go:84] Shutting down OpenAPI AggregationController I0319 12:15:51.152335 11 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0319 12:15:51.152222 11 autoregister_controller.go:168] Shutting down autoregister controller I0319 12:15:51.152407 11 controller.go:176] quota evaluator worker shutdown I0319 12:15:51.152392 11 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 12:15:51.152338 11 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0319 12:15:51.152228 11 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0319 12:15:51.152526 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" E0319 12:15:51.152596 11 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=13637&timeout=7m44s&timeoutSeconds=464&watch=true" auditID="b93a3c7c-21c4-454f-bc5f-790ae76caafd" I0319 12:15:51.152619 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 12:15:51.152649 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0319 12:15:51.152680 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0319 12:15:51.152629 11 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0319 12:15:51.152650 11 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 12:15:51.152661 11 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 12:15:51.152540 11 controller.go:176] quota evaluator worker shutdown I0319 12:15:51.152242 11 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0319 12:15:51.152248 11 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0319 12:15:51.152345 11 controller.go:176] quota evaluator worker shutdown I0319 12:15:51.152125 11 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0319 12:15:51.152841 11 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0319 12:15:51.152671 11 controller.go:86] Shutting down OpenAPI V3 AggregationController I0319 12:15:51.152688 11 gc_controller.go:91] Shutting down apiserver lease garbage collector I0319 12:15:51.152930 11 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0319 12:15:51.152716 11 controller.go:132] Ending legacy_token_tracking_controller I0319 12:15:51.152945 11 controller.go:133] Shutting down legacy_token_tracking_controller I0319 12:15:51.152729 11 system_namespaces_controller.go:76] Shutting down system namespaces controller I0319 12:15:51.152745 11 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0319 12:15:51.152755 11 local_available_controller.go:172] Shutting down LocalAvailability controller I0319 12:15:51.152888 11 controller.go:176] quota evaluator worker shutdown I0319 12:15:51.152990 11 secure_serving.go:258] Stopped listening on [::]:6443 I0319 12:15:51.153010 11 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0319 12:15:51.153024 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0319 12:15:53.153980 11 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0319 12:15:53.154037 11 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0319 12:15:53.162467 11 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0319 12:15:53.164383 11 cacher.go:171] Terminating all watchers from cacher secrets W0319 12:15:53.165386 11 cacher.go:171] Terminating all watchers from cacher endpoints W0319 12:15:53.166671 11 cacher.go:171] Terminating all watchers from cacher namespaces W0319 12:15:53.168493 11 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0319 12:15:53.171181 11 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0319 12:15:53.171731 11 cacher.go:171] Terminating all watchers from cacher podtemplates W0319 12:15:53.172316 11 cacher.go:171] Terminating all watchers from cacher nodes W0319 12:15:53.181697 11 cacher.go:171] Terminating all watchers from cacher limitranges W0319 12:15:53.182660 11 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0319 12:15:53.183133 11 cacher.go:171] Terminating all watchers from cacher resourcequotas W0319 12:15:53.183575 11 cacher.go:171] Terminating all watchers from cacher services W0319 12:15:53.186830 11 cacher.go:171] Terminating all watchers from cacher configmaps W0319 12:15:53.208896 11 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0319 12:15:53.214506 11 cacher.go:171] Terminating all watchers from cacher pods W0319 12:15:53.216111 11 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0319 12:15:53.219761 11 cacher.go:171] Terminating all watchers from cacher jobs.batch W0319 12:15:53.221381 11 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0319 12:15:53.223229 11 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0319 12:15:53.225475 11 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0319 12:15:53.226484 11 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0319 12:15:53.227400 11 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0319 12:15:53.227949 11 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0319 12:15:53.231826 11 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0319 12:15:53.232508 11 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0319 12:15:53.234277 11 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0319 12:15:53.235429 11 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0319 12:15:53.236369 11 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0319 12:15:53.237335 11 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0319 12:15:53.238750 11 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0319 12:15:53.241167 11 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0319 12:15:53.241586 11 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0319 12:15:53.242600 11 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0319 12:15:53.243887 11 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0319 12:15:53.244503 11 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0319 12:15:53.245077 11 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0319 12:15:53.246672 11 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0319 12:15:53.247229 11 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0319 12:15:53.247846 11 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0319 12:15:53.248753 11 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0319 12:15:53.249530 11 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0319 12:15:53.250338 11 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0319 12:15:53.252021 11 cacher.go:171] Terminating all watchers from cacher deployments.apps W0319 12:15:53.254136 11 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0319 12:15:53.254530 11 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0319 12:15:53.255715 11 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0319 12:15:53.256855 11 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0319 12:15:53.258395 11 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0319 12:15:53.259447 11 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0319 12:15:53.259730 11 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0319 12:15:53.261341 11 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0319 12:15:53.261723 11 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0319 12:15:53.262524 11 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0319 12:15:53.263229 11 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0319 12:15:53.263954 11 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0319 12:15:53.264469 11 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0319 12:15:53.264831 11 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0319 12:15:53.265273 11 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0319 12:15:53.265664 11 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0319 12:15:53.267654 11 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0319 12:15:53.268339 11 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0319 12:15:53.282316 11 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0319 12:15:53.282652 11 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0319 12:15:53.283196 11 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0319 12:15:53.284230 11 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0319 12:15:53.285360 11 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0319 12:15:53.285899 11 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0319 12:15:53.286381 11 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0319 12:15:53.287067 11 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0319 12:15:53.287514 11 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0319 12:15:53.288387 11 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0319 12:15:53.289049 11 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0319 12:15:53.289583 11 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0319 12:15:53.290308 11 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0319 12:15:53.291388 11 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0319 12:15:53.291764 11 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0319 12:15:53.292603 11 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0319 12:15:53.293439 11 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0319 12:15:53.294127 11 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0319 12:15:53.294589 11 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0319 12:15:53.295482 11 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0319 12:15:53.296606 11 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0319 12:15:53.297039 11 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0319 12:15:53.297530 11 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0319 12:15:53.297957 11 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0319 12:15:53.298350 11 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0319 12:15:53.298751 11 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0319 12:15:53.299118 11 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 12:15:53.299384 11 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 12:15:53.299672 11 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0319 12:15:53.301040 11 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0319 12:15:53.301551 11 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0319 12:15:53.302077 11 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0319 12:15:53.302731 11 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0319 12:15:53.303179 11 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0319 12:15:53.304268 11 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0319 12:15:53.304937 11 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0319 12:15:53.305472 11 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0319 12:15:53.306158 11 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0319 12:15:53.306896 11 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0319 12:15:53.307221 11 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0319 12:15:53.307591 11 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0319 12:15:53.308372 11 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0319 12:15:53.308666 11 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0319 12:15:53.309466 11 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0319 12:15:53.309828 11 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0319 12:15:53.310780 11 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0319 12:15:53.311206 11 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0319 12:15:53.311686 11 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0319 12:15:53.312152 11 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0319 12:15:53.312743 11 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0319 12:15:53.313310 11 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0319 12:15:53.314253 11 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0319 12:15:53.314644 11 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0319 12:15:53.315222 11 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0319 12:15:53.315823 11 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0319 12:15:53.316672 11 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0319 12:15:53.317504 11 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0319 12:15:53.319010 11 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0319 12:15:53.319548 11 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0319 12:15:53.321436 11 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0319 12:15:53.324569 11 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0319 12:15:53.325004 11 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0319 12:15:53.325410 11 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0319 12:15:53.325729 11 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0319 12:15:53.326375 11 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0319 12:15:53.326900 11 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0319 12:15:53.327371 11 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0319 12:15:53.328516 11 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0319 12:15:53.329026 11 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0319 12:15:53.329577 11 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0319 12:15:53.330564 11 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0319 12:15:53.331416 11 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0319 12:15:53.332828 11 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0319 12:15:53.333209 11 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0319 12:15:53.334621 11 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0319 12:15:53.335399 11 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0319 12:15:53.335721 11 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0319 12:15:53.336145 11 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0319 12:15:53.336699 11 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0319 12:15:53.338338 11 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0319 12:15:53.338675 11 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0319 12:15:53.339210 11 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0319 12:15:53.339632 11 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0319 12:15:53.340254 11 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0319 12:15:53.340775 11 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0319 12:15:53.341492 11 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0319 12:15:53.342285 11 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0319 12:15:53.342757 11 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0319 12:15:53.343620 11 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0319 12:15:53.343994 11 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0319 12:15:53.347023 11 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0319 12:15:53.347495 11 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0319 12:15:53.347894 11 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0319 12:15:53.348260 11 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0319 12:15:53.348720 11 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0319 12:15:53.349249 11 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0319 12:15:53.349661 11 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0319 12:15:53.349978 11 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0319 12:15:53.350580 11 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0319 12:15:53.351061 11 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0319 12:15:53.351542 11 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0319 12:15:53.352209 11 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0319 12:15:53.352579 11 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0319 12:15:53.353209 11 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0319 12:15:53.353827 11 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0319 12:15:53.354369 11 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0319 12:15:53.354722 11 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0319 12:15:53.355170 11 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0319 12:15:53.355632 11 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0319 12:15:53.356441 11 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0319 12:15:53.357032 11 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0319 12:15:53.357555 11 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0319 12:15:53.358138 11 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0319 12:15:53.360102 11 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0319 12:15:53.360416 11 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0319 12:15:53.360880 11 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0319 12:15:53.361708 11 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0319 12:15:53.362404 11 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0319 12:15:53.362741 11 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0319 12:15:53.363394 11 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0319 12:15:53.363982 11 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0319 12:15:53.364408 11 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io I0319 12:15:53.398189 1 main.go:235] Termination finished with exit code 0 I0319 12:15:53.398333 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" --- I0319 12:19:28.944525 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0319 12:19:28.944733 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0319 12:19:28.944740 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0319 12:19:28.944815 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0319 12:19:29.060413 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0319 12:19:29.060462 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0319 12:19:29.094023 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0319 12:19:29.231597 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0319 12:19:29.231631 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0319 12:19:29.231888 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0319 12:19:29.231910 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0319 12:19:29.232051 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0319 12:19:29.282908 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0319 12:19:29.282941 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0319 12:19:29.283024 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0319 12:19:29.283096 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0319 12:19:29.283124 12 controller.go:157] Shutting down quota evaluator I0319 12:19:29.283136 12 controller.go:176] quota evaluator worker shutdown I0319 12:19:29.284415 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0319 12:19:29.284444 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller E0319 12:19:29.284684 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=16705&timeout=5m50s&timeoutSeconds=350&watch=true" auditID="15b115a8-83ca-4e32-9434-ee59ee0ded8b" I0319 12:19:29.285247 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 12:19:29.285492 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0319 12:19:29.285536 12 controller.go:120] Shutting down OpenAPI V3 controller I0319 12:19:29.285554 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0319 12:19:29.285573 12 controller.go:170] Shutting down OpenAPI controller I0319 12:19:29.285586 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0319 12:19:29.285603 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0319 12:19:29.285616 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0319 12:19:29.285628 12 naming_controller.go:305] Shutting down NamingConditionController I0319 12:19:29.285642 12 establishing_controller.go:92] Shutting down EstablishingController I0319 12:19:29.285655 12 autoregister_controller.go:168] Shutting down autoregister controller I0319 12:19:29.285674 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0319 12:19:29.285691 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0319 12:19:29.285741 12 controller.go:132] Ending legacy_token_tracking_controller I0319 12:19:29.285751 12 controller.go:133] Shutting down legacy_token_tracking_controller I0319 12:19:29.285767 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0319 12:19:29.285778 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0319 12:19:29.285788 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0319 12:19:29.285819 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0319 12:19:29.285838 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0319 12:19:29.285846 12 controller.go:176] quota evaluator worker shutdown I0319 12:19:29.285898 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 12:19:29.285924 12 controller.go:176] quota evaluator worker shutdown I0319 12:19:29.285931 12 controller.go:176] quota evaluator worker shutdown I0319 12:19:29.285926 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0319 12:19:29.285937 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0319 12:19:29.285950 12 controller.go:176] quota evaluator worker shutdown I0319 12:19:29.285955 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0319 12:19:29.285971 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 12:19:29.285993 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0319 12:19:29.286023 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0319 12:19:29.286037 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0319 12:19:29.286054 12 controller.go:84] Shutting down OpenAPI AggregationController I0319 12:19:29.286100 12 secure_serving.go:258] Stopped listening on [::]:6443 I0319 12:19:29.286119 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0319 12:19:29.286039 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0319 12:19:29.286133 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0319 12:19:29.286372 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 12:19:29.286398 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 12:19:29.286416 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0319 12:19:29.286907 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 12:19:31.286703 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0319 12:19:31.286746 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0319 12:19:31.289372 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0319 12:19:31.290533 12 cacher.go:171] Terminating all watchers from cacher endpoints W0319 12:19:31.293866 12 cacher.go:171] Terminating all watchers from cacher secrets W0319 12:19:31.295917 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0319 12:19:31.297920 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0319 12:19:31.298435 12 cacher.go:171] Terminating all watchers from cacher limitranges W0319 12:19:31.298983 12 cacher.go:171] Terminating all watchers from cacher pods W0319 12:19:31.301331 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0319 12:19:31.301732 12 cacher.go:171] Terminating all watchers from cacher services W0319 12:19:31.308543 12 cacher.go:171] Terminating all watchers from cacher nodes W0319 12:19:31.309604 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0319 12:19:31.311180 12 cacher.go:171] Terminating all watchers from cacher namespaces W0319 12:19:31.314474 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0319 12:19:31.316894 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0319 12:19:31.318426 12 cacher.go:171] Terminating all watchers from cacher configmaps W0319 12:19:31.323873 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0319 12:19:31.324718 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0319 12:19:31.325367 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0319 12:19:31.340548 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0319 12:19:31.344400 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0319 12:19:31.345029 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0319 12:19:31.345740 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0319 12:19:31.346408 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0319 12:19:31.348275 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0319 12:19:31.348984 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0319 12:19:31.349846 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0319 12:19:31.350761 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0319 12:19:31.353534 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0319 12:19:31.355023 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0319 12:19:31.359615 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0319 12:19:31.360959 12 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0319 12:19:31.361479 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0319 12:19:31.362073 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0319 12:19:31.362639 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0319 12:19:31.363490 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0319 12:19:31.364547 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0319 12:19:31.365416 12 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0319 12:19:31.366098 12 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0319 12:19:31.367113 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0319 12:19:31.370373 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0319 12:19:31.371930 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0319 12:19:31.372571 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0319 12:19:31.372960 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0319 12:19:31.374576 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0319 12:19:31.375227 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0319 12:19:31.375573 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0319 12:19:31.376093 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0319 12:19:31.377107 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0319 12:19:31.378901 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0319 12:19:31.379249 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0319 12:19:31.379820 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0319 12:19:31.380753 12 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0319 12:19:31.381408 12 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0319 12:19:31.381869 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0319 12:19:31.382510 12 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0319 12:19:31.382870 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0319 12:19:31.383539 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0319 12:19:31.384914 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0319 12:19:31.385401 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0319 12:19:31.387847 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0319 12:19:31.388459 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0319 12:19:31.389363 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0319 12:19:31.390187 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0319 12:19:31.390695 12 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0319 12:19:31.391345 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0319 12:19:31.392328 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0319 12:19:31.392928 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0319 12:19:31.393751 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0319 12:19:31.394203 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0319 12:19:31.394752 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0319 12:19:31.395293 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0319 12:19:31.395946 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0319 12:19:31.396672 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0319 12:19:31.397520 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0319 12:19:31.398341 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0319 12:19:31.399453 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0319 12:19:31.400488 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0319 12:19:31.401391 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0319 12:19:31.402146 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0319 12:19:31.402741 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0319 12:19:31.403769 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0319 12:19:31.404193 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0319 12:19:31.404709 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0319 12:19:31.405326 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0319 12:19:31.406352 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0319 12:19:31.406943 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0319 12:19:31.407813 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0319 12:19:31.408399 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0319 12:19:31.409288 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0319 12:19:31.411335 12 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0319 12:19:31.411735 12 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0319 12:19:31.412295 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0319 12:19:31.413109 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0319 12:19:31.414120 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0319 12:19:31.414520 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0319 12:19:31.414948 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0319 12:19:31.415622 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0319 12:19:31.416357 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0319 12:19:31.416969 12 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0319 12:19:31.417672 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0319 12:19:31.418556 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0319 12:19:31.420245 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0319 12:19:31.420826 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0319 12:19:31.421451 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0319 12:19:31.422541 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0319 12:19:31.423631 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0319 12:19:31.425596 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0319 12:19:31.426159 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0319 12:19:31.426897 12 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0319 12:19:31.427549 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0319 12:19:31.427995 12 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0319 12:19:31.428415 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0319 12:19:31.428836 12 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0319 12:19:31.429709 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0319 12:19:31.430248 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0319 12:19:31.430729 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0319 12:19:31.431304 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0319 12:19:31.431747 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0319 12:19:31.432410 12 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0319 12:19:31.432852 12 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0319 12:19:31.433303 12 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0319 12:19:31.433586 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0319 12:19:31.433888 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0319 12:19:31.434193 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0319 12:19:31.434551 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0319 12:19:31.434891 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0319 12:19:31.435373 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0319 12:19:31.436146 12 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0319 12:19:31.436673 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0319 12:19:31.437282 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0319 12:19:31.438211 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0319 12:19:31.439381 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0319 12:19:31.440916 12 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0319 12:19:31.441529 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0319 12:19:31.442447 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0319 12:19:31.443140 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0319 12:19:31.443740 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0319 12:19:31.444206 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0319 12:19:31.444626 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0319 12:19:31.445612 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0319 12:19:31.446382 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0319 12:19:31.446734 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0319 12:19:31.447153 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0319 12:19:31.447781 12 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0319 12:19:31.448367 12 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0319 12:19:31.448766 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0319 12:19:31.449521 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0319 12:19:31.450038 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0319 12:19:31.451729 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 12:19:31.452212 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 12:19:31.452630 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0319 12:19:31.453205 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0319 12:19:31.453722 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0319 12:19:31.454341 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0319 12:19:31.455313 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0319 12:19:31.455717 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0319 12:19:31.456282 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0319 12:19:31.456755 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0319 12:19:31.457690 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0319 12:19:31.458166 12 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0319 12:19:31.458664 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0319 12:19:31.460706 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0319 12:19:31.461482 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0319 12:19:31.462427 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0319 12:19:31.463112 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0319 12:19:31.463447 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0319 12:19:31.463957 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0319 12:19:31.464368 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0319 12:19:31.464754 12 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0319 12:19:31.465184 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0319 12:19:31.466164 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0319 12:19:31.467068 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0319 12:19:31.467496 12 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0319 12:19:31.467925 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0319 12:19:31.468499 12 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0319 12:19:31.468922 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0319 12:19:31.469349 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0319 12:19:31.469690 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0319 12:19:31.470127 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0319 12:19:31.470513 12 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0319 12:19:31.471062 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0319 12:19:31.472105 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0319 12:19:31.472712 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io I0319 12:19:31.501080 1 main.go:235] Termination finished with exit code 0 I0319 12:19:31.501128 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" ---