--- I0319 09:28:45.066256 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0319 09:28:45.067238 9 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0319 09:28:45.067284 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0319 09:28:45.067640 9 controller.go:128] Shutting down kubernetes service endpoint reconciler I0319 09:28:45.120866 9 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0319 09:28:45.120906 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0319 09:28:45.129505 9 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0319 09:28:45.173588 9 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0319 09:28:45.173640 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0319 09:28:45.173991 9 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0319 09:28:45.174084 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0319 09:28:45.174043 9 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0319 09:28:45.217416 9 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0319 09:28:45.217494 9 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0319 09:28:45.217581 9 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0319 09:28:45.217661 9 controller.go:157] Shutting down quota evaluator I0319 09:28:45.217675 9 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 09:28:45.217689 9 controller.go:176] quota evaluator worker shutdown I0319 09:28:45.217701 9 controller.go:176] quota evaluator worker shutdown I0319 09:28:45.217708 9 controller.go:176] quota evaluator worker shutdown I0319 09:28:45.217716 9 controller.go:176] quota evaluator worker shutdown I0319 09:28:45.217723 9 controller.go:176] quota evaluator worker shutdown I0319 09:28:45.217799 9 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0319 09:28:45.218113 9 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0319 09:28:45.218133 9 establishing_controller.go:92] Shutting down EstablishingController I0319 09:28:45.218170 9 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0319 09:28:45.218183 9 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0319 09:28:45.218199 9 controller.go:132] Ending legacy_token_tracking_controller I0319 09:28:45.218207 9 controller.go:133] Shutting down legacy_token_tracking_controller I0319 09:28:45.218217 9 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0319 09:28:45.218232 9 gc_controller.go:91] Shutting down apiserver lease garbage collector I0319 09:28:45.218246 9 crd_finalizer.go:281] Shutting down CRDFinalizer I0319 09:28:45.218251 9 local_available_controller.go:172] Shutting down LocalAvailability controller I0319 09:28:45.218268 9 system_namespaces_controller.go:76] Shutting down system namespaces controller I0319 09:28:45.218277 9 controller.go:170] Shutting down OpenAPI controller I0319 09:28:45.218280 9 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0319 09:28:45.218296 9 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0319 09:28:45.218300 9 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0319 09:28:45.218112 9 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0319 09:28:45.218122 9 naming_controller.go:305] Shutting down NamingConditionController I0319 09:28:45.218173 9 controller.go:120] Shutting down OpenAPI V3 controller I0319 09:28:45.218208 9 autoregister_controller.go:168] Shutting down autoregister controller I0319 09:28:45.218302 9 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0319 09:28:45.219761 9 controller.go:84] Shutting down OpenAPI AggregationController I0319 09:28:45.219816 9 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0319 09:28:45.219917 9 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0319 09:28:45.219946 9 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 09:28:45.219962 9 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 09:28:45.220000 9 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0319 09:28:45.220027 9 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0319 09:28:45.220045 9 controller.go:86] Shutting down OpenAPI V3 AggregationController I0319 09:28:45.220097 9 secure_serving.go:258] Stopped listening on [::]:6443 I0319 09:28:45.220118 9 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0319 09:28:45.220135 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0319 09:28:45.220188 9 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0319 09:28:45.221026 9 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0319 09:28:45.221059 9 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 09:28:45.221077 9 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0319 09:28:45.221093 9 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 09:28:45.221122 9 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 09:28:45.221406 9 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" E0319 09:28:45.225673 9 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=10598&timeout=5m27s&timeoutSeconds=327&watch=true" auditID="7e5b8cc4-c92b-472c-a9ec-65885677ff15" I0319 09:28:47.220879 9 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0319 09:28:47.220942 9 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0319 09:28:47.225312 9 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0319 09:28:47.226633 9 cacher.go:171] Terminating all watchers from cacher services W0319 09:28:47.239120 9 cacher.go:171] Terminating all watchers from cacher nodes W0319 09:28:47.245598 9 cacher.go:171] Terminating all watchers from cacher configmaps W0319 09:28:47.274421 9 cacher.go:171] Terminating all watchers from cacher podtemplates W0319 09:28:47.277726 9 cacher.go:171] Terminating all watchers from cacher secrets W0319 09:28:47.302959 9 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0319 09:28:47.305220 9 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0319 09:28:47.307515 9 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0319 09:28:47.308361 9 cacher.go:171] Terminating all watchers from cacher endpoints W0319 09:28:47.309740 9 cacher.go:171] Terminating all watchers from cacher namespaces W0319 09:28:47.311915 9 cacher.go:171] Terminating all watchers from cacher resourcequotas W0319 09:28:47.313689 9 cacher.go:171] Terminating all watchers from cacher limitranges W0319 09:28:47.314662 9 cacher.go:171] Terminating all watchers from cacher pods W0319 09:28:47.322646 9 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0319 09:28:47.323451 9 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0319 09:28:47.324965 9 cacher.go:171] Terminating all watchers from cacher jobs.batch W0319 09:28:47.325754 9 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0319 09:28:47.327065 9 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0319 09:28:47.327644 9 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0319 09:28:47.328159 9 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0319 09:28:47.328864 9 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0319 09:28:47.330328 9 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0319 09:28:47.339727 9 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0319 09:28:47.341335 9 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0319 09:28:47.342259 9 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0319 09:28:47.343264 9 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0319 09:28:47.347850 9 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0319 09:28:47.350754 9 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0319 09:28:47.352519 9 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0319 09:28:47.357987 9 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0319 09:28:47.360702 9 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0319 09:28:47.361643 9 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0319 09:28:47.362352 9 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0319 09:28:47.363374 9 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0319 09:28:47.364184 9 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0319 09:28:47.366448 9 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0319 09:28:47.376707 9 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0319 09:28:47.378734 9 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0319 09:28:47.379764 9 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0319 09:28:47.380980 9 cacher.go:171] Terminating all watchers from cacher deployments.apps W0319 09:28:47.382861 9 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0319 09:28:47.384101 9 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0319 09:28:47.385363 9 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0319 09:28:47.386455 9 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0319 09:28:47.387690 9 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0319 09:28:47.388655 9 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0319 09:28:47.390723 9 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0319 09:28:47.392176 9 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0319 09:28:47.392977 9 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0319 09:28:47.393644 9 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0319 09:28:47.394521 9 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0319 09:28:47.395629 9 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0319 09:28:47.396357 9 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0319 09:28:47.397385 9 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0319 09:28:47.398135 9 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0319 09:28:47.399144 9 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0319 09:28:47.399889 9 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0319 09:28:47.402615 9 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0319 09:28:47.403088 9 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0319 09:28:47.403959 9 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0319 09:28:47.407313 9 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io W0319 09:28:47.407800 9 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0319 09:28:47.408391 9 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0319 09:28:47.409341 9 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0319 09:28:47.410362 9 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0319 09:28:47.411341 9 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0319 09:28:47.412261 9 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0319 09:28:47.413361 9 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0319 09:28:47.416092 9 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0319 09:28:47.417053 9 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0319 09:28:47.419978 9 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0319 09:28:47.421713 9 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0319 09:28:47.424333 9 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0319 09:28:47.425208 9 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0319 09:28:47.425935 9 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0319 09:28:47.427752 9 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0319 09:28:47.429307 9 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0319 09:28:47.430087 9 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0319 09:28:47.430982 9 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0319 09:28:47.431765 9 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0319 09:28:47.432848 9 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0319 09:28:47.433584 9 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0319 09:28:47.434320 9 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0319 09:28:47.436203 9 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0319 09:28:47.437120 9 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0319 09:28:47.439788 9 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0319 09:28:47.440434 9 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0319 09:28:47.451790 9 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0319 09:28:47.460638 9 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0319 09:28:47.462180 9 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0319 09:28:47.463910 9 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0319 09:28:47.465661 9 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0319 09:28:47.466361 9 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0319 09:28:47.467529 9 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0319 09:28:47.468161 9 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0319 09:28:47.469389 9 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0319 09:28:47.470515 9 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0319 09:28:47.471289 9 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0319 09:28:47.471912 9 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0319 09:28:47.472779 9 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0319 09:28:47.473553 9 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0319 09:28:47.474148 9 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0319 09:28:47.474719 9 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0319 09:28:47.475330 9 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0319 09:28:47.475997 9 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0319 09:28:47.479244 9 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 09:28:47.480444 9 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 09:28:47.481048 9 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0319 09:28:47.481434 9 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0319 09:28:47.482218 9 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0319 09:28:47.482964 9 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0319 09:28:47.483416 9 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0319 09:28:47.484977 9 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0319 09:28:47.486062 9 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0319 09:28:47.486689 9 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0319 09:28:47.487323 9 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0319 09:28:47.488125 9 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0319 09:28:47.488917 9 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0319 09:28:47.489820 9 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0319 09:28:47.491920 9 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0319 09:28:47.494964 9 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0319 09:28:47.495423 9 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0319 09:28:47.495984 9 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0319 09:28:47.496669 9 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0319 09:28:47.498485 9 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0319 09:28:47.499205 9 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0319 09:28:47.499699 9 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0319 09:28:47.502335 9 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0319 09:28:47.502883 9 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0319 09:28:47.503576 9 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0319 09:28:47.503924 9 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0319 09:28:47.504742 9 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0319 09:28:47.505376 9 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0319 09:28:47.506112 9 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0319 09:28:47.506648 9 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0319 09:28:47.507290 9 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0319 09:28:47.507796 9 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0319 09:28:47.508816 9 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0319 09:28:47.509695 9 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0319 09:28:47.510204 9 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0319 09:28:47.510638 9 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0319 09:28:47.512239 9 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0319 09:28:47.513030 9 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0319 09:28:47.513963 9 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0319 09:28:47.514302 9 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0319 09:28:47.515202 9 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0319 09:28:47.515673 9 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0319 09:28:47.516161 9 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0319 09:28:47.516719 9 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0319 09:28:47.517515 9 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0319 09:28:47.518026 9 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0319 09:28:47.518627 9 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0319 09:28:47.519154 9 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0319 09:28:47.519570 9 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0319 09:28:47.520243 9 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0319 09:28:47.520694 9 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0319 09:28:47.521435 9 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0319 09:28:47.521991 9 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0319 09:28:47.522581 9 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0319 09:28:47.524510 9 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0319 09:28:47.524990 9 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0319 09:28:47.525519 9 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0319 09:28:47.526023 9 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0319 09:28:47.526579 9 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0319 09:28:47.526931 9 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0319 09:28:47.527434 9 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0319 09:28:47.528152 9 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0319 09:28:47.529180 9 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0319 09:28:47.530020 9 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0319 09:28:47.530850 9 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0319 09:28:47.532985 9 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0319 09:28:47.533632 9 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0319 09:28:47.534231 9 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0319 09:28:47.535140 9 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0319 09:28:47.536171 9 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0319 09:28:47.536898 9 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0319 09:28:47.537291 9 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0319 09:28:47.538194 9 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0319 09:28:47.539263 9 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0319 09:28:47.539784 9 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0319 09:28:47.540775 9 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0319 09:28:47.541915 9 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io I0319 09:28:47.583518 1 main.go:235] Termination finished with exit code 0 I0319 09:28:47.583866 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" --- I0319 09:36:44.737154 1 main.go:213] Received signal terminated. Forwarding to sub-process "hyperkube". I0319 09:36:44.737623 12 genericapiserver.go:554] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0319 09:36:44.737631 12 controller.go:128] Shutting down kubernetes service endpoint reconciler I0319 09:36:44.737691 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ShutdownInitiated' Received signal to terminate, becoming unready, but keeping serving I0319 09:36:44.749004 12 genericapiserver.go:544] "[graceful-termination] shutdown event" name="AfterShutdownDelayDuration" I0319 09:36:44.749053 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'AfterShutdownDelayDuration' The minimal shutdown duration of 0s finished W0319 09:36:44.757879 12 lease.go:265] Resetting endpoints for master service "kubernetes" to [] I0319 09:36:44.771572 12 genericapiserver.go:721] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0319 09:36:44.771640 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationPreShutdownHooksFinished' All pre-shutdown hooks have been finished I0319 09:36:44.772000 12 genericapiserver.go:633] "[graceful-termination] shutdown event" name="NotAcceptingNewRequest" I0319 09:36:44.772033 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'InFlightRequestsDrained' All non long-running request(s) in-flight have drained I0319 09:36:44.772212 12 genericapiserver.go:679] "[graceful-termination] not going to wait for active watch request(s) to drain" I0319 09:36:44.811773 12 genericapiserver.go:670] [graceful-termination] in-flight non long-running request(s) have drained I0319 09:36:44.811816 12 genericapiserver.go:711] "[graceful-termination] shutdown event" name="InFlightRequestsDrained" I0319 09:36:44.811834 12 controller.go:157] Shutting down quota evaluator I0319 09:36:44.811853 12 controller.go:176] quota evaluator worker shutdown I0319 09:36:44.811878 12 genericapiserver.go:735] "[graceful-termination] audit backend shutdown completed" I0319 09:36:44.811937 12 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0319 09:36:44.811947 12 controller.go:176] quota evaluator worker shutdown I0319 09:36:44.811960 12 controller.go:176] quota evaluator worker shutdown I0319 09:36:44.811966 12 controller.go:176] quota evaluator worker shutdown I0319 09:36:44.811972 12 controller.go:176] quota evaluator worker shutdown I0319 09:36:44.812254 12 apf_controller.go:389] Shutting down API Priority and Fairness config worker I0319 09:36:44.812337 12 controller.go:132] Ending legacy_token_tracking_controller I0319 09:36:44.812378 12 apiaccess_count_controller.go:95] Shutting down APIRequestCount controller. I0319 09:36:44.812384 12 controller.go:133] Shutting down legacy_token_tracking_controller I0319 09:36:44.812421 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 09:36:44.812410 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 09:36:44.812514 12 dynamic_serving_content.go:149] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.crt::/etc/kubernetes/static-pod-certs/secrets/aggregator-client/tls.key" I0319 09:36:44.812423 12 apiservice_controller.go:134] Shutting down APIServiceRegistrationController I0319 09:36:44.812555 12 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/static-pod-certs/configmaps/client-ca/ca-bundle.crt" I0319 09:36:44.812428 12 apiapproval_controller.go:201] Shutting down KubernetesAPIApprovalPolicyConformantConditionController I0319 09:36:44.812679 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/external-loadbalancer-serving-certkey/tls.key" I0319 09:36:44.812697 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.crt::/etc/kubernetes/static-pod-resources/secrets/localhost-recovery-serving-certkey/tls.key" I0319 09:36:44.812706 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/localhost-serving-cert-certkey/tls.key" I0319 09:36:44.812724 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" I0319 09:36:44.812728 12 dynamic_serving_content.go:149] "Shutting down controller" name="sni-serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 09:36:44.812398 12 autoregister_controller.go:168] Shutting down autoregister controller I0319 09:36:44.812765 12 gc_controller.go:91] Shutting down apiserver lease garbage collector I0319 09:36:44.812764 12 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.crt::/etc/kubernetes/static-pod-certs/secrets/service-network-serving-certkey/tls.key" I0319 09:36:44.812415 12 crd_finalizer.go:281] Shutting down CRDFinalizer I0319 09:36:44.812819 12 secure_serving.go:258] Stopped listening on [::]:6443 I0319 09:36:44.812839 12 genericapiserver.go:624] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0319 09:36:44.812869 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HTTPServerStoppedListening' HTTP Server has stopped listening I0319 09:36:44.812935 12 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0319 09:36:44.812437 12 nonstructuralschema_controller.go:207] Shutting down NonStructuralSchemaConditionController I0319 09:36:44.812984 12 dynamic_cafile_content.go:175] "Shutting down controller" name="request-header::/etc/kubernetes/static-pod-certs/configmaps/aggregator-client-ca/ca-bundle.crt" I0319 09:36:44.812446 12 local_available_controller.go:172] Shutting down LocalAvailability controller I0319 09:36:44.812447 12 naming_controller.go:305] Shutting down NamingConditionController E0319 09:36:44.813197 12 wrap.go:53] "Timeout or abort while handling" logger="UnhandledError" method="GET" URI="/apis/user.openshift.io/v1/groups?allowWatchBookmarks=true&resourceVersion=15106&timeout=8m26s&timeoutSeconds=506&watch=true" auditID="046ff3e0-c101-42ee-b14e-d34725c3ced7" I0319 09:36:44.812453 12 controller.go:86] Shutting down OpenAPI V3 AggregationController I0319 09:36:44.812456 12 establishing_controller.go:92] Shutting down EstablishingController I0319 09:36:44.812401 12 crdregistration_controller.go:146] Shutting down crd-autoregister controller I0319 09:36:44.812532 12 controller.go:84] Shutting down OpenAPI AggregationController I0319 09:36:44.812537 12 cluster_authentication_trust_controller.go:466] Shutting down cluster_authentication_trust_controller controller I0319 09:36:44.812548 12 controller.go:120] Shutting down OpenAPI V3 controller I0319 09:36:44.812554 12 storage_flowcontrol.go:186] APF bootstrap ensurer is exiting I0319 09:36:44.812547 12 clusterquotamapping.go:142] Shutting down ClusterQuotaMappingController controller I0319 09:36:44.812560 12 customresource_discovery_controller.go:328] Shutting down DiscoveryController I0319 09:36:44.812575 12 controller.go:170] Shutting down OpenAPI controller I0319 09:36:44.812661 12 remote_available_controller.go:449] Shutting down RemoteAvailability controller I0319 09:36:44.812670 12 system_namespaces_controller.go:76] Shutting down system namespaces controller I0319 09:36:46.813349 12 genericapiserver.go:742] [graceful-termination] apiserver is exiting I0319 09:36:46.813400 12 patch_genericapiserver.go:97] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-master-0", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'TerminationGracefulTerminationFinished' All pending requests processed W0319 09:36:46.819157 12 cacher.go:171] Terminating all watchers from cacher apiservices.apiregistration.k8s.io W0319 09:36:46.820676 12 cacher.go:171] Terminating all watchers from cacher secrets W0319 09:36:46.822967 12 cacher.go:171] Terminating all watchers from cacher configmaps W0319 09:36:46.833760 12 cacher.go:171] Terminating all watchers from cacher persistentvolumeclaims W0319 09:36:46.839950 12 cacher.go:171] Terminating all watchers from cacher nodes W0319 09:36:46.842619 12 cacher.go:171] Terminating all watchers from cacher limitranges W0319 09:36:46.843087 12 cacher.go:171] Terminating all watchers from cacher pods W0319 09:36:46.845330 12 cacher.go:171] Terminating all watchers from cacher services W0319 09:36:46.861341 12 cacher.go:171] Terminating all watchers from cacher resourcequotas W0319 09:36:46.862077 12 cacher.go:171] Terminating all watchers from cacher endpoints W0319 09:36:46.864923 12 cacher.go:171] Terminating all watchers from cacher namespaces W0319 09:36:46.866345 12 cacher.go:171] Terminating all watchers from cacher serviceaccounts W0319 09:36:46.867761 12 cacher.go:171] Terminating all watchers from cacher persistentvolumes W0319 09:36:46.868894 12 cacher.go:171] Terminating all watchers from cacher replicationcontrollers W0319 09:36:46.871357 12 cacher.go:171] Terminating all watchers from cacher podtemplates W0319 09:36:46.871827 12 cacher.go:171] Terminating all watchers from cacher horizontalpodautoscalers.autoscaling W0319 09:36:46.872757 12 cacher.go:171] Terminating all watchers from cacher jobs.batch W0319 09:36:46.873430 12 cacher.go:171] Terminating all watchers from cacher cronjobs.batch W0319 09:36:46.874072 12 cacher.go:171] Terminating all watchers from cacher certificatesigningrequests.certificates.k8s.io W0319 09:36:46.874577 12 cacher.go:171] Terminating all watchers from cacher leases.coordination.k8s.io W0319 09:36:46.874829 12 cacher.go:171] Terminating all watchers from cacher endpointslices.discovery.k8s.io W0319 09:36:46.875580 12 cacher.go:171] Terminating all watchers from cacher networkpolicies.networking.k8s.io W0319 09:36:46.876041 12 cacher.go:171] Terminating all watchers from cacher ingresses.networking.k8s.io W0319 09:36:46.876548 12 cacher.go:171] Terminating all watchers from cacher ingressclasses.networking.k8s.io W0319 09:36:46.877075 12 cacher.go:171] Terminating all watchers from cacher runtimeclasses.node.k8s.io W0319 09:36:46.877891 12 cacher.go:171] Terminating all watchers from cacher poddisruptionbudgets.policy W0319 09:36:46.879808 12 cacher.go:171] Terminating all watchers from cacher roles.rbac.authorization.k8s.io W0319 09:36:46.881864 12 cacher.go:171] Terminating all watchers from cacher rolebindings.rbac.authorization.k8s.io W0319 09:36:46.883388 12 cacher.go:171] Terminating all watchers from cacher clusterroles.rbac.authorization.k8s.io W0319 09:36:46.884527 12 cacher.go:171] Terminating all watchers from cacher clusterrolebindings.rbac.authorization.k8s.io W0319 09:36:46.885081 12 cacher.go:171] Terminating all watchers from cacher priorityclasses.scheduling.k8s.io W0319 09:36:46.885583 12 cacher.go:171] Terminating all watchers from cacher storageclasses.storage.k8s.io W0319 09:36:46.886899 12 cacher.go:171] Terminating all watchers from cacher volumeattachments.storage.k8s.io W0319 09:36:46.888594 12 cacher.go:171] Terminating all watchers from cacher csinodes.storage.k8s.io W0319 09:36:46.889353 12 cacher.go:171] Terminating all watchers from cacher csidrivers.storage.k8s.io W0319 09:36:46.890372 12 cacher.go:171] Terminating all watchers from cacher csistoragecapacities.storage.k8s.io W0319 09:36:46.892061 12 cacher.go:171] Terminating all watchers from cacher flowschemas.flowcontrol.apiserver.k8s.io W0319 09:36:46.892637 12 cacher.go:171] Terminating all watchers from cacher prioritylevelconfigurations.flowcontrol.apiserver.k8s.io W0319 09:36:46.893120 12 cacher.go:171] Terminating all watchers from cacher controllerrevisions.apps W0319 09:36:46.893814 12 cacher.go:171] Terminating all watchers from cacher deployments.apps W0319 09:36:46.896933 12 cacher.go:171] Terminating all watchers from cacher daemonsets.apps W0319 09:36:46.898993 12 cacher.go:171] Terminating all watchers from cacher replicasets.apps W0319 09:36:46.899998 12 cacher.go:171] Terminating all watchers from cacher statefulsets.apps W0319 09:36:46.901258 12 cacher.go:171] Terminating all watchers from cacher validatingwebhookconfigurations.admissionregistration.k8s.io W0319 09:36:46.902566 12 cacher.go:171] Terminating all watchers from cacher mutatingwebhookconfigurations.admissionregistration.k8s.io W0319 09:36:46.903923 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicies.admissionregistration.k8s.io W0319 09:36:46.904281 12 cacher.go:171] Terminating all watchers from cacher validatingadmissionpolicybindings.admissionregistration.k8s.io W0319 09:36:46.906556 12 cacher.go:171] Terminating all watchers from cacher customresourcedefinitions.apiextensions.k8s.io W0319 09:36:46.907325 12 cacher.go:171] Terminating all watchers from cacher adminnetworkpolicies.policy.networking.k8s.io W0319 09:36:46.907787 12 cacher.go:171] Terminating all watchers from cacher featuregates.config.openshift.io W0319 09:36:46.910570 12 cacher.go:171] Terminating all watchers from cacher ipaddresses.ipam.cluster.x-k8s.io W0319 09:36:46.910989 12 cacher.go:171] Terminating all watchers from cacher hostupdatepolicies.metal3.io W0319 09:36:46.911419 12 cacher.go:171] Terminating all watchers from cacher clustercsidrivers.operator.openshift.io W0319 09:36:46.911954 12 cacher.go:171] Terminating all watchers from cacher authentications.operator.openshift.io W0319 09:36:46.912636 12 cacher.go:171] Terminating all watchers from cacher machineosconfigs.machineconfiguration.openshift.io W0319 09:36:46.913497 12 cacher.go:171] Terminating all watchers from cacher bmceventsubscriptions.metal3.io W0319 09:36:46.913981 12 cacher.go:171] Terminating all watchers from cacher controlplanemachinesets.machine.openshift.io W0319 09:36:46.914258 12 cacher.go:171] Terminating all watchers from cacher installplans.operators.coreos.com W0319 09:36:46.914799 12 cacher.go:171] Terminating all watchers from cacher projecthelmchartrepositories.helm.openshift.io W0319 09:36:46.915595 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 09:36:46.916387 12 cacher.go:171] Terminating all watchers from cacher alertmanagerconfigs.monitoring.coreos.com W0319 09:36:46.916917 12 cacher.go:171] Terminating all watchers from cacher olmconfigs.operators.coreos.com W0319 09:36:46.917557 12 cacher.go:171] Terminating all watchers from cacher alertingrules.monitoring.openshift.io W0319 09:36:46.917828 12 cacher.go:171] Terminating all watchers from cacher consolelinks.console.openshift.io W0319 09:36:46.918170 12 cacher.go:171] Terminating all watchers from cacher catalogsources.operators.coreos.com W0319 09:36:46.919152 12 cacher.go:171] Terminating all watchers from cacher insightsoperators.operator.openshift.io W0319 09:36:46.919762 12 cacher.go:171] Terminating all watchers from cacher prometheuses.monitoring.coreos.com W0319 09:36:46.920521 12 cacher.go:171] Terminating all watchers from cacher tuneds.tuned.openshift.io W0319 09:36:46.921507 12 cacher.go:171] Terminating all watchers from cacher clusterextensions.olm.operatorframework.io W0319 09:36:46.922150 12 cacher.go:171] Terminating all watchers from cacher machines.machine.openshift.io W0319 09:36:46.922711 12 cacher.go:171] Terminating all watchers from cacher kubeletconfigs.machineconfiguration.openshift.io W0319 09:36:46.923942 12 cacher.go:171] Terminating all watchers from cacher operatorconditions.operators.coreos.com W0319 09:36:46.924746 12 cacher.go:171] Terminating all watchers from cacher kubecontrollermanagers.operator.openshift.io W0319 09:36:46.926179 12 cacher.go:171] Terminating all watchers from cacher performanceprofiles.performance.openshift.io W0319 09:36:46.926818 12 cacher.go:171] Terminating all watchers from cacher dnsrecords.ingress.operator.openshift.io W0319 09:36:46.928848 12 cacher.go:171] Terminating all watchers from cacher consoleclidownloads.console.openshift.io W0319 09:36:46.929741 12 cacher.go:171] Terminating all watchers from cacher machineautoscalers.autoscaling.openshift.io W0319 09:36:46.931226 12 cacher.go:171] Terminating all watchers from cacher subscriptions.operators.coreos.com W0319 09:36:46.932313 12 cacher.go:171] Terminating all watchers from cacher ippools.whereabouts.cni.cncf.io W0319 09:36:46.932920 12 cacher.go:171] Terminating all watchers from cacher machineconfigpools.machineconfiguration.openshift.io W0319 09:36:46.934161 12 cacher.go:171] Terminating all watchers from cacher hostfirmwarecomponents.metal3.io W0319 09:36:46.934738 12 cacher.go:171] Terminating all watchers from cacher consoleplugins.console.openshift.io W0319 09:36:46.935768 12 cacher.go:171] Terminating all watchers from cacher configs.samples.operator.openshift.io W0319 09:36:46.936361 12 cacher.go:171] Terminating all watchers from cacher images.config.openshift.io W0319 09:36:46.937266 12 cacher.go:171] Terminating all watchers from cacher thanosrulers.monitoring.coreos.com W0319 09:36:46.937846 12 cacher.go:171] Terminating all watchers from cacher baremetalhosts.metal3.io W0319 09:36:46.938759 12 cacher.go:171] Terminating all watchers from cacher storages.operator.openshift.io W0319 09:36:46.939444 12 cacher.go:171] Terminating all watchers from cacher storagestates.migration.k8s.io W0319 09:36:46.940124 12 cacher.go:171] Terminating all watchers from cacher openshiftapiservers.operator.openshift.io W0319 09:36:46.940688 12 cacher.go:171] Terminating all watchers from cacher nodeslicepools.whereabouts.cni.cncf.io W0319 09:36:46.941502 12 cacher.go:171] Terminating all watchers from cacher egressqoses.k8s.ovn.org W0319 09:36:46.941965 12 cacher.go:171] Terminating all watchers from cacher proxies.config.openshift.io W0319 09:36:46.943682 12 cacher.go:171] Terminating all watchers from cacher clusterautoscalers.autoscaling.openshift.io W0319 09:36:46.944767 12 cacher.go:171] Terminating all watchers from cacher metal3remediations.infrastructure.cluster.x-k8s.io W0319 09:36:46.945305 12 cacher.go:171] Terminating all watchers from cacher adminpolicybasedexternalroutes.k8s.ovn.org W0319 09:36:46.945913 12 cacher.go:171] Terminating all watchers from cacher machinehealthchecks.machine.openshift.io W0319 09:36:46.946346 12 cacher.go:171] Terminating all watchers from cacher imagetagmirrorsets.config.openshift.io W0319 09:36:46.947060 12 cacher.go:171] Terminating all watchers from cacher consolenotifications.console.openshift.io W0319 09:36:46.947629 12 cacher.go:171] Terminating all watchers from cacher clusteruserdefinednetworks.k8s.ovn.org W0319 09:36:46.948134 12 cacher.go:171] Terminating all watchers from cacher configs.imageregistry.operator.openshift.io W0319 09:36:46.948880 12 cacher.go:171] Terminating all watchers from cacher hostfirmwaresettings.metal3.io W0319 09:36:46.949343 12 cacher.go:171] Terminating all watchers from cacher apiservers.config.openshift.io W0319 09:36:46.949986 12 cacher.go:171] Terminating all watchers from cacher schedulers.config.openshift.io W0319 09:36:46.950273 12 cacher.go:171] Terminating all watchers from cacher consoleyamlsamples.console.openshift.io W0319 09:36:46.950915 12 cacher.go:171] Terminating all watchers from cacher preprovisioningimages.metal3.io W0319 09:36:46.951387 12 cacher.go:171] Terminating all watchers from cacher userdefinednetworks.k8s.ovn.org W0319 09:36:46.951960 12 cacher.go:171] Terminating all watchers from cacher consolequickstarts.console.openshift.io W0319 09:36:46.952403 12 cacher.go:171] Terminating all watchers from cacher rolebindingrestrictions.authorization.openshift.io W0319 09:36:46.952939 12 cacher.go:171] Terminating all watchers from cacher dnses.config.openshift.io W0319 09:36:46.953415 12 cacher.go:171] Terminating all watchers from cacher cloudcredentials.operator.openshift.io W0319 09:36:46.953970 12 cacher.go:171] Terminating all watchers from cacher clusteroperators.config.openshift.io W0319 09:36:46.955334 12 cacher.go:171] Terminating all watchers from cacher machineconfigurations.operator.openshift.io W0319 09:36:46.958264 12 cacher.go:171] Terminating all watchers from cacher machineosbuilds.machineconfiguration.openshift.io W0319 09:36:46.959040 12 cacher.go:171] Terminating all watchers from cacher egressrouters.network.operator.openshift.io W0319 09:36:46.959586 12 cacher.go:171] Terminating all watchers from cacher configs.operator.openshift.io W0319 09:36:46.960378 12 cacher.go:171] Terminating all watchers from cacher firmwareschemas.metal3.io W0319 09:36:46.960931 12 cacher.go:171] Terminating all watchers from cacher consoles.config.openshift.io W0319 09:36:46.961936 12 cacher.go:171] Terminating all watchers from cacher volumesnapshots.snapshot.storage.k8s.io W0319 09:36:46.963219 12 cacher.go:171] Terminating all watchers from cacher kubeapiservers.operator.openshift.io W0319 09:36:46.964082 12 cacher.go:171] Terminating all watchers from cacher egressfirewalls.k8s.ovn.org W0319 09:36:46.964833 12 cacher.go:171] Terminating all watchers from cacher controllerconfigs.machineconfiguration.openshift.io W0319 09:36:46.965318 12 cacher.go:171] Terminating all watchers from cacher prometheusrules.monitoring.coreos.com W0319 09:36:46.965772 12 cacher.go:171] Terminating all watchers from cacher csisnapshotcontrollers.operator.openshift.io W0319 09:36:46.966387 12 cacher.go:171] Terminating all watchers from cacher consolesamples.console.openshift.io W0319 09:36:46.966857 12 cacher.go:171] Terminating all watchers from cacher consoles.operator.openshift.io W0319 09:36:46.967855 12 cacher.go:171] Terminating all watchers from cacher metal3remediationtemplates.infrastructure.cluster.x-k8s.io W0319 09:36:46.968295 12 cacher.go:171] Terminating all watchers from cacher ingresses.config.openshift.io W0319 09:36:46.969502 12 cacher.go:171] Terminating all watchers from cacher olms.operator.openshift.io W0319 09:36:46.970208 12 cacher.go:171] Terminating all watchers from cacher ipamclaims.k8s.cni.cncf.io W0319 09:36:46.971186 12 cacher.go:171] Terminating all watchers from cacher networks.operator.openshift.io W0319 09:36:46.971787 12 cacher.go:171] Terminating all watchers from cacher rangeallocations.security.internal.openshift.io W0319 09:36:46.972154 12 cacher.go:171] Terminating all watchers from cacher imagecontentpolicies.config.openshift.io W0319 09:36:46.972670 12 cacher.go:171] Terminating all watchers from cacher machineconfigs.machineconfiguration.openshift.io W0319 09:36:46.973438 12 cacher.go:171] Terminating all watchers from cacher ingresscontrollers.operator.openshift.io W0319 09:36:46.974859 12 cacher.go:171] Terminating all watchers from cacher operatorgroups.operators.coreos.com W0319 09:36:46.976215 12 cacher.go:171] Terminating all watchers from cacher imagepruners.imageregistry.operator.openshift.io W0319 09:36:46.976655 12 cacher.go:171] Terminating all watchers from cacher egressips.k8s.ovn.org W0319 09:36:46.977030 12 cacher.go:171] Terminating all watchers from cacher securitycontextconstraints.security.openshift.io W0319 09:36:46.977983 12 cacher.go:171] Terminating all watchers from cacher authentications.config.openshift.io W0319 09:36:46.978516 12 cacher.go:171] Terminating all watchers from cacher oauths.config.openshift.io W0319 09:36:46.979146 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotcontents.snapshot.storage.k8s.io W0319 09:36:46.979984 12 cacher.go:171] Terminating all watchers from cacher nodes.config.openshift.io W0319 09:36:46.980541 12 cacher.go:171] Terminating all watchers from cacher kubeschedulers.operator.openshift.io W0319 09:36:46.980907 12 cacher.go:171] Terminating all watchers from cacher provisionings.metal3.io W0319 09:36:46.981297 12 cacher.go:171] Terminating all watchers from cacher machinesets.machine.openshift.io W0319 09:36:46.981791 12 cacher.go:171] Terminating all watchers from cacher operatorhubs.config.openshift.io W0319 09:36:46.982277 12 cacher.go:171] Terminating all watchers from cacher clusterversions.config.openshift.io W0319 09:36:46.984006 12 cacher.go:171] Terminating all watchers from cacher clustercatalogs.olm.operatorframework.io W0319 09:36:46.984602 12 cacher.go:171] Terminating all watchers from cacher podmonitors.monitoring.coreos.com W0319 09:36:46.985119 12 cacher.go:171] Terminating all watchers from cacher probes.monitoring.coreos.com W0319 09:36:46.985733 12 cacher.go:171] Terminating all watchers from cacher hardwaredata.metal3.io W0319 09:36:46.986172 12 cacher.go:171] Terminating all watchers from cacher profiles.tuned.openshift.io W0319 09:36:46.986613 12 cacher.go:171] Terminating all watchers from cacher operatorpkis.network.operator.openshift.io W0319 09:36:46.987283 12 cacher.go:171] Terminating all watchers from cacher clusterserviceversions.operators.coreos.com W0319 09:36:46.988101 12 cacher.go:171] Terminating all watchers from cacher dataimages.metal3.io W0319 09:36:46.988498 12 cacher.go:171] Terminating all watchers from cacher alertrelabelconfigs.monitoring.openshift.io W0319 09:36:46.988976 12 cacher.go:171] Terminating all watchers from cacher builds.config.openshift.io W0319 09:36:46.989867 12 cacher.go:171] Terminating all watchers from cacher ipaddressclaims.ipam.cluster.x-k8s.io W0319 09:36:46.990346 12 cacher.go:171] Terminating all watchers from cacher clusterresourcequotas.quota.openshift.io W0319 09:36:46.990784 12 cacher.go:171] Terminating all watchers from cacher servicecas.operator.openshift.io W0319 09:36:46.991227 12 cacher.go:171] Terminating all watchers from cacher network-attachment-definitions.k8s.cni.cncf.io W0319 09:36:46.991909 12 cacher.go:171] Terminating all watchers from cacher dnses.operator.openshift.io W0319 09:36:46.992327 12 cacher.go:171] Terminating all watchers from cacher consoleexternalloglinks.console.openshift.io W0319 09:36:46.992751 12 cacher.go:171] Terminating all watchers from cacher operators.operators.coreos.com W0319 09:36:46.993347 12 cacher.go:171] Terminating all watchers from cacher overlappingrangeipreservations.whereabouts.cni.cncf.io W0319 09:36:46.993728 12 cacher.go:171] Terminating all watchers from cacher baselineadminnetworkpolicies.policy.networking.k8s.io W0319 09:36:46.994132 12 cacher.go:171] Terminating all watchers from cacher imagedigestmirrorsets.config.openshift.io W0319 09:36:46.994819 12 cacher.go:171] Terminating all watchers from cacher servicemonitors.monitoring.coreos.com W0319 09:36:46.995683 12 cacher.go:171] Terminating all watchers from cacher networks.config.openshift.io W0319 09:36:46.996785 12 cacher.go:171] Terminating all watchers from cacher projects.config.openshift.io W0319 09:36:46.997438 12 cacher.go:171] Terminating all watchers from cacher imagecontentsourcepolicies.operator.openshift.io W0319 09:36:46.998120 12 cacher.go:171] Terminating all watchers from cacher storageversionmigrations.migration.k8s.io W0319 09:36:46.999022 12 cacher.go:171] Terminating all watchers from cacher volumesnapshotclasses.snapshot.storage.k8s.io W0319 09:36:46.999915 12 cacher.go:171] Terminating all watchers from cacher helmchartrepositories.helm.openshift.io W0319 09:36:47.000654 12 cacher.go:171] Terminating all watchers from cacher egressservices.k8s.ovn.org W0319 09:36:47.001039 12 cacher.go:171] Terminating all watchers from cacher kubestorageversionmigrators.operator.openshift.io W0319 09:36:47.001408 12 cacher.go:171] Terminating all watchers from cacher containerruntimeconfigs.machineconfiguration.openshift.io W0319 09:36:47.001849 12 cacher.go:171] Terminating all watchers from cacher infrastructures.config.openshift.io W0319 09:36:47.004614 12 cacher.go:171] Terminating all watchers from cacher openshiftcontrollermanagers.operator.openshift.io W0319 09:36:47.005576 12 cacher.go:171] Terminating all watchers from cacher alertmanagers.monitoring.coreos.com W0319 09:36:47.006526 12 cacher.go:171] Terminating all watchers from cacher credentialsrequests.cloudcredential.openshift.io W0319 09:36:47.007010 12 cacher.go:171] Terminating all watchers from cacher etcds.operator.openshift.io W0319 09:36:47.007749 12 cacher.go:171] Terminating all watchers from cacher apirequestcounts.apiserver.openshift.io I0319 09:36:47.055670 1 main.go:235] Termination finished with exit code 0 I0319 09:36:47.055733 1 main.go:188] Deleting termination lock file "/var/log/kube-apiserver/.terminating" ---